Wandering Thoughts archives

A little modern Unix twitch

Every so often, I just want to read a file (or a bunch of files) without doing anything to them. I have all sorts of reasons for this; sometimes I want to prime the OS's disk cache, or I want to time the file read speed, or I want to put some IO load on the system, or any number of other reasons (the worst is to just update the file atimes). The common element is that I don't care what happens to the file data after it gets read off the disk, so I dump it in /dev/null.

When I do this these days, I never do the redirection to /dev/null in the same process that is doing the reading; instead, I always feed things through a pipe. In other words, instead of running:

cat file >/dev/null

I run:

cat file | cat >/dev/null

This is completely wasteful and annoying, but the problem with not doing this is that far too many commands and Unix systems are too smart for their own good these days; there are all sorts of things that notice you are writing to /dev/null and optimize away all of that read IO that I want to happen. Putting a pipe in the middle kills all of those optimizations because no matter how optimized the writer is, the data has to go across the pipe which means that the reader has to actually read it.

Sometimes this is unnecessary paranoia, but it's easier to be paranoid and slightly inefficient all of the time than to try to remember when I can be completely efficient and when I can't be. (It's not as if an extra cat process really matters on any modern system.)

Sidebar: How this optimization can happen naturally

I can't swear that I'm remembering something that actually happened in a real Unix, but here's an example of how this sort of stuff can get optimized without any individual component being too crazy. You need two pieces:

• a version of cat that prefers to work by mmap()'ing each source file and then write()'ing it to the output in one go.

  This is less absurd than it sounds; when mmap() was first introduced, a lot of people became very enthused about using it on everything (which sometimes led to fun bugs when these programs were asked to work on something that couldn't be mmap()'d). You can even argue that this version of cat is better because it doesn't try to guess the right buffer size, it just defers everything to the operating system.

  (If the kernel has bits of the file in kernel buffers, it can even do 'zero copy' IO, where it doesn't have to copy things to user level on a read() only to immediately copy them back into the kernel on the following write().)

• a kernel that optimizes write()'s to /dev/null by not actually copying data from the user level process into the kernel only to then discard it; instead, it just checks that the buffer given to it is a valid one and then returns immediate success.

When the file is mmap()'d, nothing is immediately read from disk; instead the reads will happen when the mapped pages are touched and produce virtual memory faults. If you wrote to a real file, this would happen when the write() started copying data from your process into kernel buffers; however, because the write() to /dev/null never does this copy, it never causes any page faults on the mapped pages and thus never does any IO to read the source file. Ergo, 'cat file >/dev/null' does nothing real and runs startlingly fast.

It's hard to argue with either of these optimizations in isolation (apart from the whole issue of hitting everything with the mmap() stick), but when they combine together you get an unfortunate result.

Unix programs should avoid exiting non-zero for clever reasons

Dan Astoorian's comment on yesterday's entry reminded me of this general issue, which I will phrase this way:

Dear authors of Unix programs, please don't invent clever reasons to have your programs exit with a non-zero status.

Unless you have a really good reason, your program should only ever exit non-zero if it has actually failed. Trust me; shell programmers are perfectly competent to check for some property of your output ourselves if we really want to know.

(Even if you think you have a really good reason and a really important non-error thing to communicate with your non-zero exit status, you will make shell programmers very happy if you provide a command line switch to turn this behavior off so that you really truly only exit non-zero for real errors. The only exception if pretty much all possible uses of your program will have to examine your exit code to determine what to do next.)

The poster child for an overly clever non-zero exit status is expr, which will exit with status 1 if your expression evaluates to zero or to an empty string. This is not just silly, it is surplus; test (aka [) gives shell script authors a perfectly good way of testing the result of our expr expressions if we actually want this feature.

I am sure that a few shell script authors in the world have found this convenient over the years, but I am equally sure that a lot more shell script authors have wound up grimly guarding expr executions in various ways precisely because of this. Or with subtle bugs in their scripts because they were not guarding against the condition, because quick, who remembers that little expr landmine?

(Both have happened to us.)

/u, one of our long-standing Unix customizations

One of Unix's small problems is that it has no simple and universal way to refer to people's home directories, especially other people's home directories. Some people are about to pipe up about ~user, but that's not a universal way; support for it has to be added to every program that will use it (and in every context where they resolve file names), and inevitably there are programs where it is not supported at all or is not supported everywhere. Beyond that, it's the wrong solution for the same reason that handling globbing in each program instead of in shells is the wrong solution.

(You can see how non-universal ~user is these days by trying it in various graphical file open dialogs. And in fact there are places where ~user support simply cannot be added, such as #! lines in shell scripts; no Unix kernel is ever going to support '#!~cks/bin/dog'.)

If you want a universal solution to a problem related to filenames and paths in Unix, there is only one way you can do it: you have to put something in the filesystem, because that's the only common thing used by absolutely every program that uses files. And you want to put it in the filesystem because this instantly makes it accessible to everything that uses files, which on a Unix system is a lot of things.

(This is the same core idea behind /proc and so on, and behind Plan 9 sticking everything in the filesystem. The filesystem is the common Unix namespace, so if you want to expose something widely, you put it in the filesystem.)

So our solution to this problem is to have a directory called /u, which is set up so that /u/<user> gives you the user's real home directory regardless of just where it is. Our implementation uses symlinks because that's the simple approach for us, but you could equally well use the automounter. The name /u is picked partly because it is short and partly because it is relatively memorable.

Having a real /u directory turns out to have a host of practical benefits, just as you'd expect on a Unix system. Here's a couple. First, it is truly universal; in anything that refers to files, in any context, you now have convenient short way of referring to anyone's home directory, yours included. You no longer have to worry about whether a particular program supports ~, ~user, or $HOME; you just use /u/<user>.

(And yes, this includes #! lines in shell scripts, so you can have scripts that do '#!/u/cks/bin/python3' if you need to.)

Second, we've wound up rewriting user home directories in /etc/passwd to use this form, because doing so persuades any number of programs to embed this form in their automatically created user configuration files, which in turn means that they no longer throw a spanner if we move a user from one filesystem to another (Firefox used to be infamous for this). Arguably any program doing this is buggy, but we're pragmatists; doing this eliminates the bug for any program that is not excessively clever.

(Excessively clever programs will try to determine the real path of the user's home directory, which can be done in our implementation.)

PS: if you are going to do this, it turns out to be convenient to do this for all entries in /etc/passwd (or at least all entries with real home directories), not just the accounts of actual users.