Wandering Thoughts archives

test is surprisingly smart

Via Hacker News I would up reading Common shell script mistakes. When I read this, I initially thought that it contained well-intentioned but mistaken advice about test (aka '[ ... ]'). Then I actually checked what test's behavior is and got a bunch of surprised. It turns out that test is really quite smart, sometimes disturbingly so.

Here's two different versions of a test expression:

[ x"$var" = x"find" ] && echo yes
[ "$var" = "find" ] && echo yes

In theory, the reason the first version has an 'x' in front of both sides is to deal with the case where someone sets $var to something that is a valid test operator, like '-a' or '-x' or even '('; after all, '[ -a = find ]' doesn't look like a valid test expression. But if you actually check, it turns out that the second version works perfectly well too.

What's going on is that test is much smarter than you might think. Rather than simply processing its arguments left to right, it uses a much more complicated process of actually parsing its command line. When I started writing this entry I thought it was just modern versions that behaved this way, but in fact the behavior is much older than that; it goes all the way back to the V7 version of test, which actually implements a little recursive descent parser (in quite readable code). This behavior is even specified in the Single Unix Specification page for test where you can read the gory details for yourself (well, most of them).

(The exception is that the SuS version of test doesn't include -a for and or -o for or. This is an interesting exclusion since it turns out they were actually in the V7 version of test per eg the manpage.)

Note that this cleverness can break down in extreme situations. For example, '[ "$var1" -a "$var2" -a "$var3" ]' is potentially dangerous; consider what happens if $var2 is '-r'. And of course you still really want to use "..." to force things to be explicit empty arguments, because an outright missing argument can easily completely change the meaning of a test expression. Consider what happens to '[ -r $var ]' if $var is empty.

(It reduces to '[ -r ]', which is true because -r is not the empty string. You probably intended it to be false because a zero-length file name is considered unreadable.)

The difference between no argument and an empty argument

Here is a little Bourne shell quiz. Supposing that $VAR is not defined, are the following two lines equivalent?

./acnt $VAR
./acnt "$VAR"

The answer is no. If the acnt script is basically 'echo "$#"', then the first one will print 0 and the second one will print 1; in other word, the first line called acnt with no argument and the second one called acnt with one argument (that happens to be an empty string).

Unix shells almost universally draw some sort of a distinction between a variable expansion that results in no argument and an empty argument (although they can vary in how you force an empty argument). This is what we're seeing here; in the Bourne shell, using a "..." forces there to always be a single argument regardless of what $VAR expands to or doesn't. Sometimes this is useful behavior, for example when it means that a program is invoked with exactly a specific number of arguments (and with certain things in certain argument positions) even if some things aren't there. Sometimes this is inconvenient, if what you really wanted was to quote $VAR but not necessarily pass acnt an empty argument if $VAR wound up unset. If you want this latter behavior, you need to use the more awkward form:

./acnt ${VAR:+"$VAR"}

(Support for this is required by the Single Unix Specification and is present in Solaris 10, so I think you're very likely to find it everywhere.)

Note that it can be possible to indirectly notice the presence of empty arguments in situations where they don't show directly. For example:

$ echo a "$VAR" b
a  b

If you look carefully there is an extra space printed between a and b here; that is because echo is actually printing 'a', separator space, an empty string, another separator space, and then 'b'. Of course some programs are more obvious, even if the error message is a bit more mysterious:

$ cat "$VAR"
cat: : No such file or directory

(This entry is brought to you in the process of me discovering something interesting about modern versions of test, but that's another entry.)

Unix getopt versus Google's getopt variant and why Unix getopt wins

The longer title of this entry is 'why Google's version of getopt is right for Google but wrong for (almost) everyone else'.

One reaction to my rant about Go's getopt problem is to ask what the big problem is. Looked at in the right light, the rules implemented by Go's flag package (which are apparently the Google more or less standard for parsing flags) actually have a certain amount to recommend them because they're less ambiguous, more consistent, and less likely to surprise you. For example, consider this classic error: start with a command that takes a -t flag switch and a -f flag with an argument. One day someone in a hurry accidently writes this as 'cmd -ft fred bob' (instead of '-tf'). If you're lucky this will fail immediately with an error; if you're unlucky this quietly succeeds but does something quite different than what you expected. Google flag parsing reduces the chances of this by forcing you to always separate flags (so you can't do this just by transposing two characters).

In an environment where you are mostly or entirely using commands that parse flags this way, you get a number of benefits like this. I assume that this describes Google, which I suspect is heavily into internal tooling. But most environments are not like this at all; instead, commands using Go's flag package (or equivalents) are going to be the minority and the vast majority of commands you use will instead be either standard Unix commands or programs that use the same general argument parsing (partly because it's the default in almost everyone's standard library). In such environments the benefits that might come from Google flag parsing are dwarfed by the fact that it is just plain different from almost everything else you use. You will spend more time cursing because 'cmd -tf fred bob' gives you 'no such flag -tf' errors than you will ever likely save in the one (theoretical) time you type 'cmd -ft fred bob'.

(In theory you could also sort of solve the problem by always separating flag arguments even for programs that don't need this. But this is unlikely in practice since such programs are probably the majority of what you run and anyways, other bits of Go flag handling aren't at all compatible with standard Unix practice.)

In other words: inside Google, the de facto standard is whatever Google's tools do because you're probably mostly using them. Outside Google, the de facto standard is what the majority of programs do and that is standard Unix getopt (and extended GNU getopt). Deviations from de facto Unix getopt behavior cause the same problems that deviations from other de facto standards cause.

Now I'll admit that this is stating the case a bit too strongly. There are plenty of Unix programs that already deviate to a greater or lesser degree from standard Unix getopt. As with all interface standards a large part of what matters is how often people are going to use the deviant commands; the more frequently, the more you can get away with odd command behavior.

(You can also get away with more odd behavior if it's basically impossible to use your program casually. If an occasional user is going to have to re-read your documentation every time, well, you can (re)explain your odd command line behavior there.)

Modern versions of Unix are more adjustable than they used to be

One of the slow changes in modern Unix over the past ten to fifteen years has been a significant increase in modularity and with it how adjustable a number of core things are without major work. This has generally not been something that ordinary users notice because it happens at the level of system-wide configuration.

Undoubtedly this all sounds abstract, so let's get concrete. The first example here is the relative pervasiveness of PAM. In the pre-PAM world, implementing additional password strength checks or special custom rules for who could su to who took non-trivial modifications to the source for passwd and su (or sudo). In the modern world both are simple PAM modules, as is things like taking special custom actions when a password is changed.

My next example is nsswitch.conf. There was a day in the history of Unix when adding DNS lookups to programs required recompiling them against a library with a special version of gethostbyname() et al. These days, how any number of things get looked up is not merely something that you can configure but something you can control; if you want or need to, you can add a new sort of lookup yourself as an aftermarket do it yourself thing. This can be exploited for clever hacks that don't require changing the system's programs in any particular way, just exploiting how they work (although there are limits imposed by this approach).

(Actually now that I'm writing this entry I'm not sure that there have been any major moves in this sort of core modularity beyond NSS and PAM. Although there certainly are more options for things like your cron daemon and your syslog daemon if you feel like doing wholesale replacement of programs.)

One of the things that these changes do is they reduce the need for operating system source since they reduce your need for custom versions of operating system commands.

(Of course you can still wind up needing OS source in order to figure out how to write your PAM or NSS module.)

Sidebar: best practices have improved too

One of the practical increases in modularity has come from an increasing number of programs (such as many versions of cron) scanning directories instead of just reading a file. As we learned starting no later than BSD init versus System V init, a bunch of files in a directory is often easier to manage than a monolithic single file because you can have all sorts of people dropping files in and updating their own files without colliding with each other. Things like Linux package management have strongly encouraged this approach.

NFS's problem with (concurrent) writes

If you hang around distributed filesystem developers, you may hear them say grumpy things about NFS's handling of concurrent writes and writes in general. If you're an outsider this can be a little bit opaque. I didn't fully remember the details until I was reminded about them recently so in my usual tradition I am going to write down the core problem. To start with I should say that the core problem is with NFS the protocol, not any particular implementation.

Suppose that you have two processes, A and B. A is writing to a file and B is reading from it (perhaps they are cooperating database processes or something). If A and B are running on the same machine, the moment that A calls write() the newly-written data is visible to B when it next does a read() (or it's directly visible if B has the file mmap()'d). Now we put A and B on different machines, sharing access to the file over NFS. Suddenly we have a problem, or actually two problems.

First, NFS is silent on how long A's kernel can hold on to the write() before sending it to the NFS server. If A close()s or fsync()s the file the kernel must ship the writes off to the NFS server, but before then it may hang on to them for some amount of time at its convenience. Second, NFS has no protocol for the server to notify B's kernel that there is updated data in the file. Instead B's kernel may be holding on to what is now old cached data that it will quietly give to B, even though the server has new data. Properly functioning NFS clients check for this when you open() a file (and discard old data if necessary); I believe that they may check at other times but it's not necessarily guaranteed.

The CS way of putting this is that this is a distributed cache invalidation problem and NFS has only very basic support for it. Basically NFS punts and tells you to use higher-level mechanisms to make this work, mechanisms that mean A and B have to be at least a bit NFS-aware. Many modern distributed and cluster filesystems have much more robust support that guarantees processes A and B see a result much closer to what they would if they ran on the same machine (some distributed FSes probably guarantee that it's basically equivalent).

(Apparently one term of art for this is that NFS has only 'close to open' consistency, ie you only get consistent results among a pool of clients if A closes the file before B opens it.)

Sun's NeWS was a mistake, as are all toolkit-in-server windowing systems

One of the great white hopes of the part of the Unix world that never liked X Windows was Sun's NeWS. Never mind all of its practical flaws, all sorts of people held NeWS up as the better way and the bright future that could have been if only things had been different (by which they mean if people had made the 'right' choice instead of settling for X Windows). One of the reasons people often give for liking NeWS is that it put much of the windowing toolkit into the server instead of forcing every client to implement it separately.

Unfortunately for all of these people, history has fairly conclusively shown that NeWS was a mistake. Specifically the core design of putting as much intelligence as possible into the server instead of the clients has turned out to be a terrible idea. There are at least two big reasons for this.

The first is parallelization. In the increasingly multi-core world you desperately want as much concurrent processing as possible and it's much easier to run several clients in parallel than it is to parallelize a single server. Even if you do get equal parallelization, separate clients are inherently more resilient because the operating system intrinsically imposes a strong separation of address space and so on, something that's very hard to get in server where everything is jumbled together.

(I believe that this is one reason that modern X font rendering has been moved from the server to the client. XFT font rendering is increasingly complex and CPU-consuming, so it's better to stick clients with that burden than dump all of it on the server.)

The second is that if you put the toolkit in the server you make evolving the toolkit and its API much more complicated and problematic. The drawback of having everyone use the server toolkit is that everyone has to use the same server toolkit. Well, not completely. You can introduce a mechanism to have multiple toolkit versions and APIs all in the same server and allow clients to select which one they want or need and so on and so forth. The mess of a situation with the current X server and its extensions make a very educational example of what happens if you go down this path; not very much of it is good.

(Some X extensions are in practice mandatory but still must be probed for and negotiated by the clients, while others are basically historical relics but they still can't be dropped because some client somewhere may ask for them.)

Toolkits in the client push the burden of dealing with the evolution of the toolkit into the clients. It is clients that carry around old or new versions of the toolkit, with various different APIs, and you naturally have old toolkit versions (and even old toolkits) go away entirely when they are no longer used by any active clients (or even any installed clients, when things get far enough).

(I'm ignoring potential security issues for complex reasons, but they may be a good third reason to be unhappy with server-side toolkits.)

I am not a (Unix) purist

Someone from outside looking at my environment and what I use and do might think that I'm a Unix purist; after all, I use an odd and limited shell, a stripped down and custom desktop environment, and I persist in reading my email with MH (which is a very Unixy thing in the abstract) instead of a modern IMAP client. I could go on, but I think you get the general drift. That impression would be a mistake. I am not really a Unix purist.

Oh, I certainly have a tinge of it (along with the upturned nose that goes with it, which I try not to let show much these days). I feel genuine attraction to the ideals of Unix and much of their embodiment. But ultimately I'm a pragmatist who is interested in getting things done, not an idealist. I use all of the minimalistic, Unix-purist things that I do because they work well for me, not because I'm intrinsically opposed to their alternatives and refuse to touch them because they're impure.

One consequence of this is that I'm perfectly happy to depart from the straight and narrow path of Unix minimalism and purism when the result works better for me. I have in the past, I do right now in various ways, and I undoubtedly will do so in the future. It's just that I tend not to talk about these departures because usually they're not interesting (because non-minimalism is increasingly the default state of things on Unix, for good reason).

Sidebar: my latest non-minimal departure

As one might guess from a recent interest, I turned on filename completion and command line editing in my unusual shell. Previously I had two or three reasons for not doing this: it wasn't efficient on old machines, I thought it was impure, and it clashed badly with 9term, one of my terminal emulators. Well, old machines are long gone, a Linux kernel change broke 9term, and using bash (by default) on some systems around here had slowly sold me on the real and seductive convenience of filename completion. I found myself missing it in my regular shell then realized that now that I'm not using 9term there was no reason not to build a version with GNU Readline support.

So far I rather like it and actually think that I should have done this some years ago. (This is often how changes in my environment go.)

A slow realization: many of my dotfiles don't need to be dotfiles

Like many Unix people, I have a slowly accumulating drift of dotfiles (and dot-directories) cluttering up my $HOME. It has recently struck me that a certain amount of these dotfiles are in fact a self-inflicted injury, by which I mean that they don't actually have to be dotfiles in $HOME and I could move them elsewhere if I wanted to. There are two variants of this injury.

The first variety is where I have simply made a file a dotfile in $HOME out of reflexive habit. There is no system-supplied program that expects to find it there; I have simply needed a file for some purpose and made it a dotfile because it seemed to make sense. For example, I have a set of X resource files that I made into dotfiles basically just because. Since it's my X startup script that loads them into the X server, they could perfectly well live in, say, $HOME/lib/X11 under some set of sensible (and visible) names.

(Often there is some sort of vague link to something that was a necessary dotfile back in history.)

The second variety is where a program defaults to using a dotfile but this can be changed with a command line option and I already run the program in some automated way (through a cover script or the like). Here I can perfectly well change my automation to relocate the dotfile to some better place. There are probably a number of programs I'm running like this.

(Similar to this is things that can be moved with an environment variable, like $INPUTRC for $HOME/.inputrc.)

Certainly going forward I think my rule is going to be that I won't create any new dotfiles unless I have absolutely no choice. Relocating existing ones, well, I have a lot of inertia and my existing setup works, it's just got a massive clutter in $HOME that I can't see unless I do an 'ls -a'.

(Why dotfiles are a bad idea is probably not necessarily obvious, but that's another entry.)

PS: what brought this realization on was building out a version of my environment for Cinnamon on my laptop. When I was copying dotfiles over to the laptop and working on them I started asking myself why they actually were dotfiles instead of being organized somewhere sensible where I could see them.

The feature (or features) I really want added to xterm

I recently mentioned that there were only a few features that could attract me away from xterm. Since a commentator asked about them, today I'm going to try to talk about them.

The big feature that I would like is for widening and narrowing the terminal window to properly reflow text. In other words, if I have lines that wrapped around and I widen the terminal window I want the lines to no longer wrap around. Today if you do that you just get a big blank space on the right. Similarly if I narrow a window I don't want the lines truncated, I want them re-wrapped.

(Mac OS X terminal windows do this. I have envied them that ever since I saw it in action many years ago.)

There are technical limitations and caveats about this but it's perfectly possible to do it in many common situations (ones where the output has been simply printed to the screen instead of placed with cursor addressing). Xterm actually goes part of the way to this today in that its line selection for copy-and-paste sometimes recognizes that a logical line spans more than the physical on-screen line.

The other interesting feature is signposted by gnome-terminal doing a much more restricted version of it: g-t will recognize URLs, underline them if you hover the mouse over them, and then give you a context sensitive popup menu that lets you do things with them. I would like a generalized version of this where you can provide something like regular expressions to match and context menu entries to add for each match.

There are probably more exotic things you could add to a terminal emulator if you wanted to; for instance, there's no reason why a terminal emulator couldn't directly render pictures if asked to. I can't say I actively want such features, though, partly because I'm not sure that we have a good idea of how to use them in a Unix command line environment.

(Also, I'm not interested in giving up what I consider xterm's core features for me. Trying to list those is something for another entry.)

PS: it's likely that there are other features that would pull me away from xterm that I just haven't had the vision to imagine. I try to always remember that the really great features tend to be the ones that you didn't even realize that you wanted before you heard about them.

What's changed in Unix networking in the last decade or so

In an earlier entry I mentioned in passing that a number of things had changed in Unix networking since the classic Stevens work was written. Today I feel like trying to inventory at least some of them:

• IPv6 is growing in importance. If you care about this (and you should) there is a whole exciting world of issues with dual binding, detecting when the machine has useful IPv6, and so on. Note that real IPv6 support may require examining hidden assumptions in your code.

• along with IPv6 has come a number of new interfaces that are now the correct way of doing things, such as getaddrinfo(). There are some subtleties here that deserve to be carefully covered in any good modern networking book.

• people now care about handling a lot of connections at once in an efficient manner. This has created both new interfaces (such as poll() and epoll()) and new asynchronous server approaches.

• similarly, threading has become a big issue and there are a bunch of issues surrounding good file descriptor handling in the face of threading. Overly simple code can have any number of inobvious races where your code winds up manipulating something other than it expected because other threads have created and destroyed file descriptors behind your back.

• practical protocol design now requires considering how your new thing will interact with firewalls, which have become ubiquitous in the past decade.

• TCP congestion control and window management algorithms have evolved over the past decade in ways that affect TCP performance in real world situations.
• there is a whole area of protocol performance on the modern Internet, where you care about things like DNS lookups, keeping the sizes of things down so that you can fit them in one packet, and so on. My impression is that most of this is new in the past decade.

• at least Linux has added support for doing various interesting things over local Unix domain sockets.

Although it's not quite within the scope of a work on basic (Unix) socket network coding, I think that any new book on this should at least say 'do not attempt to design your own cryptographically secure communication protocol'. Some discussion of SSL/TLS may be in order since it's become so pervasive.