A peculiarity of the GNU Coreutils version of 'test' and '['

November 24, 2023

Famously, '[' is a program, not a piece of shell syntax, and it's also known as 'test' (which was the original name for it). On many systems, this was and is implemented by '[' being a hardlink to 'test' (generally 'test' was the primary name for various reasons). However, today I found out that GNU Coreutils is an exception. Although the two names are built from the same source code (src/test.c), they're different binaries and the '[' binary is larger than the 'test' binary. What is ultimately going on here is a piece of 'test' behavior that I had forgotten about, that of the meaning of running 'test' with a single argument.

The POSIX specification for test is straightforward. A single argument is taken as a string, and the behavior is the same as for -n, although POSIX phrases it differently:

string
True if the string string is not the null string; otherwise, false.

The problem for GNU Coreutils is that GNU programs like to support options like --help and --version. Support for these is specifically disallowed for 'test', where 'test --help' and 'test --version' must both be silently true. However, this is not disallowed by POSIX for '[' if '[' is invoked without the closing ']':

$ [ --version
[ (GNU coreutils) 9.1
[...]
$ [ foo
[: missing ‘]’
$ [ --version ] && echo true
true

As we can see here, invoking 'test' as '[' without the closing ']' as an argument is an error, and GNU Coreutils is thus allowed to interpret the results of your error however it likes, including making '[ --version' and so on work.

(There's a comment about it in test.c.)

The binary size difference is presumably because the 'test' binary omits the version and help text, along with the code to display it. But if you look at the relevant Coreutils test.c code, the relevant code isn't disabled with an #ifdef. Instead, LBRACKET is #defined to 0 when compiling the 'test' binary. So it seems that modern C compilers are doing dead code elimination on the 'if (LBRACKET) { ...}' section, which is a well established optimization, and then going on to notice that the called functions like 'usage()' are never invoked and dropping them from the binary. Possibly this is set with some special link time magic flags.

PS: This handling of a single argument for test goes all the way back to V7, where test was actually pretty smart. If I'm reading the V7 test(1) manual page correctly, this behavior was also documented.

PPS: In theory GNU Coreutils is portable and you might find it on any Unix. In practice I believe it's only really used on Linux.


Comments on this page:

By Ian Z at 2023-11-25 12:26:04:

Doesn't coreutils in fact install as a single multi-call binary with multiple links to it, similar to busybox? At least by default when you build it from pristine source?

By cks at 2023-11-25 23:01:17:

In a quick inspection of the source, Coreutils does look like it can build that way. However I didn't try to build it from pristine source because that appears to be rather complicated with a lot of moving parts. (I did build the Ubuntu 22.04 version from their package source, so I could be relatively confident the size difference wasn't something weird Ubuntu was doing when it built the binary packages.)

By Andy at 2023-11-26 05:26:37:

The GNU coding standards discourage programs changing behaviour depending on their name:

Please don’t make the behavior of a utility depend on the name used to invoke it. It is useful sometimes to make a link to a utility with a different name, and that should not change what it does.

And the Coreutils README has similar reasoning:

The ls, dir, and vdir commands are all separate executables instead of one program that checks argv[0] because people often rename these programs to things like gls, gnuls, l, etc. Renaming a program file shouldn't affect how it operates, so that people can get the behavior they want with whatever name they want.

I guess that applies to all the programs, not just the ls variants, so that's probably why test and [ are separate binaries. There is an option to put everything in one binary, but it's not hte default – you have to pass --enable-single-binary to configure (it can take values shebangs or symlinks to control how aliases are made).

(Perhaps the idea that people would want to call GNU ls gls or gnuls is part of the reason the name-independence thing is in the GNU coding standards – on non-GNU Unixes if the GNU version of a program is installed it'll probably be alongside the OS's own version.)

By Paxsali at 2023-11-26 05:38:26:

It's very funny to me reading your article about this topic.

I've identified since a few years now, that every unix should provide external binaries for invoking a) the printing of version information and b) help messages, instead of solely relying on the individual authors to provide a `--version` or `--help` clause. By providing external binaries, what I mean is that there is one canonical and more importantly safe way provided how to print such information about a binary, without the need to execute it. Essentially the forms are getting reversed, such that instead of the usual `somecommand --version` the command `version somecommand` is invoked.

I came to this conclusion when I wanted to do a simple statistical evaluation of all the version strings of all binaries on my system. What I did back then was essentially generate a list of all binary/on-disk commands and run them sequentially with the arguments `--version` and saving the output in a csv type file.

The invokation must have looked something like this (from the top of my head):

compgen -A command | parallel '{} --version'

To my great suprise, what happened after this is actually shocking.

My system broke for no apparent reason. To this day I don't understand the exact breaking point, but I can tell you the result was random data loss and the deletion of symlinks.

It is not safe to randomly execute binaries without knowing what they do. And more generally, for such simple tasks as merely printing version or help information, there shouldn't be a necesity to even execute a particular binary. I say this having known very well that not all binaries would support `--version`, since for some it's only `-v`, or `-V`, or maybe subcommands, i.e. `version`.

I was thinking maybe it's better to develop a sort of "binary format" by convention, that when followed, external binaries `version` would either print the version information contained in a binary, or nothing, if the binary doesn't contain any version information or is not of that mentioned (hypothetical, new) binary format. IBM's AIX uses the `what` command to find a so called `what-string` in any arbitrary file - just give an example of something similar.

There is another reason why `--version` and `--help` shouldn't be expected as a universal standard: they're GNU specific. Many resources regarding POSIX compliant programming suggest that the double dash options shouldn't be used, if POSIX compatibility is a goal. Of course, I myself got used to and cannot do without the GNU conventions - so this is only a small point. The positive side-effect of the reversal of invokation `version somecommand` or `version /path/to/somecommand` is it would be identical on POSIX and not-so-POSIX environments by it's generic form.

Pretty much all the examples and reason above also count for the `--help` analogy. We'd require a (on-disk) `help` binary to be invoked as `help command` (not to be confused with bash's builtin help!).

Suppose such a binary format was ratified and the (on-disk) commands `version` and `help` would be standard in every unix-like OS deployment, then what that would mean is the example you give had a clear solution as to what would be the correct behavior:

$ [ --version
[: missing ‘]’

# hypothetical invokation (this doesn't exist, yet!)
$ version [
[ (GNU coreutils) 9.1
[...]

$ [ --version ] && echo true
true

Anyhow your article/blog post reminded me very much of that, so I wanted to share.

That's all.

This comment is towards Paxsali.

Essentially the forms are getting reversed, such that instead of the usual `somecommand --version` the command `version somecommand` is invoked.

Famously, UNIX inverts just about every useful control structure the wrong way.

My system broke for no apparent reason. To this day I don't understand the exact breaking point, but I can tell you the result was random data loss and the deletion of symlinks.

I believe it.

I was thinking maybe it's better to develop a sort of "binary format" by convention, that when followed, external binaries `version` would either print the version information contained in a binary, or nothing, if the binary doesn't contain any version information or is not of that mentioned (hypothetical, new) binary format.

Here's the fundamental issue: UNIX isn't defined in terms of what it enables, but in terms of what it makes impossible. This version command, and its help command counterpart, describe an object-oriented system, and at that point it's no longer UNIX.

Written on 24 November 2023.
« Unix's 'test' program and the V7 Bourne shell
Unix shells and the current directory »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Nov 24 22:46:09 2023
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.