Wandering Thoughts archives

Unix's pipeline problem (okay, its problem with file redirection too)

In a comment on yesterday's entry, Mihai Cilidariu sensibly suggested that I not add timestamp support to my tools but instead outsource this to a separate program in a pipeline. In the process I would get general support for this and complete flexibility in the timestamp format. This is clearly and definitely the right Unix way to do this.

Unfortunately it's not a good way in practice, because of a fundamental pragmatic problem Unix has with pipelines. This is our old friend block buffering versus line buffering. A long time ago, Unix decided that many commands should change their behavior in the name of efficiency; if they wrote lines of output to a terminal you'd get each line as it was written, but if they wrote lines to anything else you'd only get it in blocks.

This is a big problem here because obviously a pipeline like 'monitor | timestamp' basically requires the monitor process to produce output a line at time in order to be useful; otherwise you'd get large blocks of lines that all had the same timestamp because they were written to the timestamp process in a block. The sudden conversion from line buffered to block buffered can also affect other sorts of pipeline usage.

It's certainly possible to create programs that don't have this problem, ones that always write a line at a time (or explicitly flush after every block of lines in a single report). But it is not the default, which means that if you write a program without thinking about it or being aware of the issue at all you wind up with a program that has this problem. In turn people like me can't assume that a random program we want to add timestamps to will do the right thing in a pipeline (or keep doing it).

(Sometimes the buffering can be an accidental property of how a program was implemented. If you first write a simple shell script that runs external commands and then rewrite it as a much better and more efficient Perl script, well, you've probably just added block buffering without realizing it.)

In the end, what all of this really does is that it chips away quietly at the Unix ideal that you can do everything with pipelines and that pipelining is the right way to do lots of stuff. Instead pipelining becomes mostly something that you do for bulk processing. If you use pipelines outside of bulk processing, sometimes it works, sometimes you need to remember odd workarounds so that it's mostly okay, and sometimes it doesn't do what you want at all. And unless you know Unix programming, why things are failing is pretty opaque (which doesn't encourage you to try doing things via pipelines).

(This is equally a potential problem with redirecting program output to files, but it usually hits most acutely with pipelines.)

What I want to have in shell (filename) completion

I've been using basic filename completion in my shell for a while now, and doing so has given me a perspective on what advanced features of this I'd find useful and which strike me as less useful. Unfortunately for me, the features that I'd find most useful are the ones that are the hardest to implement.

Put simply, the problem with basic filename completion is that any time you want to use even basic shell features like environment variables, you lose completion. Do you refer to some directories through convenience variables? Nope, not any more, because you can choose between completing a long name and not completing, say, '$ml/afilename'.

(I know, bash supports completing filenames that use environment variables. Probably zsh does as well. I'm not interested in switching to either right now.)

But environment variables are just one of the ways to shorten filenames. Two more cases are using wildcards to match unique or relatively unique subsets of a long filename and using various multi-match operators to specify several filenames in some directory or the like. Both of these would be handy to be able to do filename completion for. In fact, let's generalize that: what I'd really like is for my shell to be able to do filename completion in the face of any and all things that can appear in filenames and get expanded by the shell. Then I could combine the power of filename completion and the power of all of those handy shell operators together.

Of course this is easier said than done. I know that what I'm asking for is quite a challenging programming exercise and to some extent a design exercise once we get to the obscure features. But it sure would be handy (more handy, in my biased opinion, than a number of other completion features).

(I've never used eg bash's smart context aware autocompletion of program arguments in general, so I don't know for sure how useful I'd find it; however, my personal guess is 'not as much as full filename completion'. I'm so-so on autocompletion of program names; I suppose I do sometimes use programs with annoyingly long names, so my shell might as well support it. Again I'd rate it well below even autocompletion with environment variables, much less the full version.)

There's no portable way to turn a file descriptor read only or write only

It all started when John Regehr asked a good question in a tweet:

serious but undoubtedly stupid question: why does writing to file descriptor 0 in Linux and OS X work?

My first impulse was to say 'lazy code that starts with a general read/write file descriptor and doesn't bother to make it read only when the fd becomes a new process's standard input', but I decided to check the manual pages first. Much to my surprise it turns out that in Unix there is no portable way to turn a read/write file descriptor into a read-only or write-only one.

In theory the obvious way to do this is with fcntl(fd, F_SETFL, O_RDONLY) (or O_WRONLY as applicable). In practice, this is explicitly documented as not working on both Linux and FreeBSD; on them you're not allowed to affect the file access mode, only things like O_NONBLOCK. It's not clear if this behavior is compliant with the Single Unix Specification for fcntl(), but either way it's how a very large number of real systems behave in the field today so we're stuck with it.

This means that if you have, say, a shell, the shell cannot specifically restrict plain commands that it starts to have read-only standard input and write-only standard output and standard error. The best it can do is pass on its own stdin, stdout, and stderr, and if they were passed to the shell with full read/write permissions the shell has to pass them to your process with these permissions intact and so your process can write to fd 0. Only when the shell is making new file descriptors can it restrict them to be read only or write only, which means pipelines and file redirections.

Further, it turns out that in a fair number of cases it's natural to start out with a single read/write file descriptor (and in a few it's basically required). For one example, anything run on a pseudo-tty that was set up through openpty() will be this way, as the openpty() API only gives you a single file descriptor for the entire pty and obviously it has to be opened read/write. There are any number of other cases, so I'm not going to try to run through them all.

(At this point it may also have reached the point of backwards compatibility, due to ioctl() calls on terminals and ptys. I'm honestly not sure of the rules for what terminal ioctls need read and/or write permissions on the file descriptors, and I bet a bunch of other people aren't either. In that sort of environment, new programs that set up shells might be able to restrict fds 0, 1, and 2 to their correct modes but don't dare do so lest they break various shells and programs that have gotten away with being casual and uncertain.)

PS: If you want to see how a shell or a command's descriptors are set up, you can use lsof. The letter after the file descriptor's number will tell you if it's read, write, or u for r/w.

Understanding the (original) meaning of Unix load average

Most everyone knows the load average, and almost every system administrator knows that it's not necessarily a useful measure today. The problem is that the load average combines two measurements, as it counts both how many processes are trying to run and how many processes are currently waiting for IO to finish. This means that a machine having a big load average tells you very little by itself; do you have a lot of processes using the CPU, a lot of processes doing IO, a few processes doing very slow IO, or perhaps a bunch of processes waiting for an NFS server to come back to life?

As it happens, I think there is an explanation for what the load average is supposed to mean and originally did mean, back in the early days of Unix. To put it simply, it's how soon your process would get to run.

To see how this makes sense, let's rewind time to the Vaxes that 3BSD ran on when load average was added to Unix. On those machines, two things were true: in CPU-relative terms IO was faster than it is now, and the CPU was simply slow in general so that doing anything much took appreciable compute time. This means that a process waiting on 'fast' disk IO is probably going to have the IO complete before you do much computation yourself and then the process's going to have to use enough CPU time to deal with the IO results that you're going to notice, even if it's doing relatively simple processing. So runnable processes are directly contending for the CPU right now and 'busy' processes in IO wait will be contending for it before you can do very much (and the kernel will soon be doing some amount of computing on their behalf). Both sorts of processes will delay yours and so merging them together in a single 'load average' figure makes sense.

This breaks down (and broke down) as CPUs became much faster in an absolute sense as well as much faster than IO. Today a process doing only basic IO processing will use only tiny amounts of CPU time and your CPU-needing process will probably hardly notice or be delayed by it. This makes the number of processes in IO wait basically meaningless as a predictor of how soon a ready process can run and how much of the CPU it'll get; you can do a lot before their slow IO completes and when it does complete they often need almost no CPU time before they go back to waiting on IO again. There's almost no chance that a 'busy' process in IO wait will block your process from getting a CPU slice.

(As a side note, including some indicator of disk load into 'load average' also makes a lot of sense in a memory-constrained environment where a great deal of what you type at your shell prompt requires reading things off disk, which is what early BSDs on Vaxes usually were. A 100% unused CPU doesn't help you if you're waiting to read the test binary in from the disk in the face of 10 other processes trying to do their own disk IO.)

When the Unix load average was added to Unix

For reasons beyond the scope of this entry itself, I recently became curious about when the concept of 'load average' first appeared in Unix. Fortunately we have the Unix tree from the Unix Heritage Society, so I can answer this question by digging through various historical Unix trees.

The answer appears to be that the concept of load average appears first in 3BSD. In the 3BSD /usr/src/cmd directory, uptime.c is allegedly dated to October 4th 1979. The 3BSD kernel source for vmsched.c already has the normal definition of load average; it counts both runnable processes and processes waiting in uninterruptible sleep (which is theoretically always a short term thing). I believe that 3BSD also marks the appearance of vmstat.

As far as I can tell, V7 Unix lacked both an uptime command and any kernel support for accumulating the information. I was going to say that V7 didn't have any way to see how loaded your system was, but it does have a basic version of iostat and the kernel kept some degree of information about things like system versus user time, as you can see from the iostat manpage.

My personal suspicion is that 3BSD grew support for keeping track of how loaded the system was (via both uptime and the more detailed vmstat) because Berkeley started using 3BSD for undergraduate computing in student labs, where you could not simply ask your colleagues if someone was running something big and could they stop for a while if so. But I don't actually know if Berkeley was using 3BSD in undergrad labs this early on or if they only started doing it a few years later with 4BSD et al.

(UCB may also have wanted to have some idea of how well their new paged virtual memory system was working in practice.)

As as side note, I do like the BUGS section of the 3BSD vmstat manual page (at the end):

So many numbers print out that its sometimes hard to figure out what to watch.

This has only become more true over time.

Unix's mistake with rm and directories

Welcome to Unix, land of:

; rm thing
rm: cannot remove 'thing': Is a directory
; rmdir thing
;

(And also rm may only be telling you half the story, because you can have your rmdir fail with 'rmdir: failed to remove 'thing': Directory not empty'. Gee thanks both of you.)

Let me be blunt here: this is Unix exercising robot logic. Unix knows perfectly well what you want to do, it's perfectly safe to do so, and yet Unix refuses to do it (or tell you the full problem) because you didn't use the right command. Rm will even remove directories if you just tell it 'rm -r thing', although this is more dangerous than rmdir.

Once upon a time rm had almost no choice but to do this because removing directories took special magic and special permissions (as '.' and '..' and the directory tree were maintained in user space). Those days are long over, and with them all of the logic that would have justified keeping this rm (mis)feature. It lingers on only as another piece of Unix fossilization.

(This restriction is not even truly Unixy; per Norman Wilson, Research Unix's 8th edition removed the restriction, so the very heart of Unix fixed this. Sadly very little from Research Unix V8, V9, and V10 ever made it out into the world.)

PS: Some people will now say that the Single Unix Specification (and POSIX) does not permit rm to behave this way. My view is 'nuts to the SUS on this'. Many parts of real Unixes are already not strictly POSIX compliant, so if you really have to have this you can add code to rm to behave in a strictly POSIX compliant mode if some environment variable is set. (This leads into another rant.)

(I will reluctantly concede that having unlink(2) still fail on directories instead of turning into rmdir(2) is probably safest, even if I don't entirely like it either. Some program is probably counting on the behavior and there's not too much reason to change it. Rm is different in part because it is used by people; unlink(2) is not directly. Yes, I'm waving my hands a bit.)

A brief history of fiddling with Unix directories

In the beginning (say V7 Unix), Unix directories were remarkably non-special. They were basically files that the kernel knew a bit about. In particular, there was no mkdir(2) system call and the . and .. entries in each directory were real directory entries (and real hardlinks), created by hand by the mkdir program. Similarly there was no rmdir() system call and rmdir directly called unlink() on dir/.., dir/., and dir itself. To avoid the possibility of users accidentally damaging the directory tree in various ways, calling link(2) and unlink(2) on directories was restricted to the superuser.

(In part to save the superuser from themselves, commands like ln and rm then generally refused to operate on directories at all, explicitly checking for 'is this a directory' and erroring out if it was. V7 rm would remove directories with 'rm -r', but it deferred to rmdir to do the actual work. Only V7 mv has special handling for directories; it knew how to actually rename them by manipulating hardlinks to them, although this only worked when mv was run by the superuser.)

It took until 4.1 BSD or so for the kernel to take over the work of creating and deleting directories, with real mkdir() and rmdir() system calls. The kernel also picked up a rename() system call at the same time, instead of requiring mv to do the work with link(2) and unlink(2) calls; this rename() also worked on directories. This was the point, not coincidentally, where BSD directories themselves became more complicated. Interestingly, even in 4.2 BSD link(2) and unlink(2) would work on directories if you were root and mknod(2) could still be used to create them (again, if you were root), although I suspect no user level programs made use of this (and certainly rm still rejected directories as before).

(As a surprising bit of trivia, it appears that the 4.2 BSD ln lacked a specific 'is the source a directory' guard and so a superuser probably could accidentally use it to make extra hardlinks to a directory, thereby doing bad things to directory tree integrity.)

To my further surprise, raw link(2) and unlink(2) continued to work on directories as late as 4.4 BSD; it was left for other Unixes to reject this outright. Since the early Linux kernel source is relatively simple to read, I can say that Linux did from very early on. Other Unixes, I have no idea about. (I assume but don't know for sure that modern *BSD derived Unixes do reject this at the kernel level.)

(I've written other entries on aspects of Unix directories and their history: 1, 2, 3, 4.)

PS: Yes, this does mean that V7 mkdir and rmdir were setuid root, as far as I know. They did do their own permission checking in a perfectly V7-appropriate way, but in general, well, you really don't want to think too hard about V7, directory creation and deletion, and concurrency races.

In general and despite what I say about it sometimes, V7 made decisions that were appropriate for its time and its job of being a minimal system on a relatively small machine that was being operated in what was ultimately a friendly environment. Delegating proper maintenance of a core filesystem property like directory tree integrity to user code may sound very wrong to us now but I'm sure it made sense at the time (and it did things like reduce the kernel size a bit).

The latest xterm versions mangle $SHELL in annoying ways

As of patch #301 (and with changes since then), the canonical version of xterm has some unfortunate behavior changes surrounding the $SHELL environment variable and how xterm interacts with it. The full details are in the xterm manpage in the OPTIONS section, but the summary is that xterm now clears or changes $SHELL if the $SHELL value is not in /etc/shells, and sometimes even if it is. As far as I can tell, the decision tree goes like this:

1. if xterm is (explicitly) running something that is in /etc/shells (as 'xterm /some/thing', not 'xterm -e /some/thing'), $SHELL will be rewritten to that thing.

2. if xterm is running anything (including running $SHELL itself via being invoked as just 'xterm') and $SHELL is not in /etc/shells but your login shell is, $SHELL will be reset to your login shell.

3. otherwise $SHELL will be removed from the environment, resulting in a shell environment with $SHELL unset. This happens even if you run plain 'xterm' and so xterm is running $SHELL.

It is difficult for me to summarize concisely how wrong this is and how many ways it can cause problems. For a start, this is a misuse of /etc/shells, per my entry on what it is and isn't; /etc/shells is in no way a complete list of all of the shells (or all of the good shells) that are in use on the system. You cannot validate the contents of $SHELL against /etc/shells because that is not what /etc/shells is there for.

This xterm change causes significant problems for anyone with their shell set to something that is not in /etc/shells, anyone using an alternate personal shell (which is not in /etc/shells for obvious reasons), any program that assumes $SHELL is always set (historically a safe assumption), and any environment that assumes $SHELL is not reset when set to something non-standard such as a captive or special purpose 'shell'.

(Not all versions of chsh restrict you to what's in /etc/shells, for that matter; some will let you set other things if you really ask them to.)

If you fall into one or more of these categories and you use xterm, you're going to need to change your environment at some point. Unfortunately it seems unlikely that this change will be reverted, so if your version of Unix updates xterm at all you're going to have it sooner or later (so far only a few Linux distributions are recent enough to have it).

PS: Perhaps this should be my cue to switch to urxvt. However my almost-default configuration of it is still just enough different from xterm to be irritating for me, although maybe I could fix that with enough customization work. For example, I really want its double-click selection behavior to exactly match xterm because that's what my reflexes expect and demand by now. See also.

PPS: Yes, I do get quite irritated at abrupt incompatible changes in the behavior of long-standing Unix programs, at least when they affect me.

The history of commercial Unix and my pragmatism

I've said in the past that I'm a Unix pragmatist instead of a purist (and in general I've wound up a pragmatist about computing in general). In thinking about it recently, it's struck me that it's probably very hard to be both an old Unix hand and a purist, so it's not entirely surprising that I've wound up this way. To put it simply, the history of Unix since the late 1980s is a history of beautiful and pure things getting utterly crushed and ground up by the world. Over and over again, ugly things won.

An incomplete list:

• beautiful CISC and RISC architectures were driven into the ground by the x86 juggernaut. Modern 64-bit x86 is kind of an okay architecture from what I understand, but 32-bit x86 has always been an ugly register-starved monster.

• beautiful server hardware and hardware designs were crushed under the PC-compatible juggernaut and its collection of ugly hacks (eg the A20 line, which is probably still being emulated by the chipset on your 2015 vintage server).

All of the Unix vendors were eventually driven into surrender on this. Oh, I think that Oracle will still sell you SPARC based hardware, but almost no one really wants it any more because it very much lost on cost and performance.

• cool, pioneering Unix companies like MIPSco and NeXT went under or were bought out. This culminated in Sun being bought by Oracle, marking the final coda on the Unix workstation dream (which had died years before if we're being honest).

• Unix itself went through a series of uglifications in the early 1990s. Sun took SunOS 3, made it SunOS 4, and then engineered a high-speed collision between SunOS 4 and System V R4 that gave us the unappealing Solaris 2. This stampeded many of the other Unix vendors into making OSF/1, which was even worse (it was based on Mach at a time when this was not a good choice, among other defects).
• Unix vendors did terrible things to software layers on top of Unix. See CORBA, DCE, Motif, and let's be honest, Sun wasn't doing much better here either for a long time.

  It would not be much of an exaggeration to say that almost every architectural decision that most Unix vendors made turned out to be a mistake.

• Plan 9 was a complete practical failure. The Unix vendors collectively turned their back on essentially everything that came from the very source of Unix itself, and so by and large did Unix users. Later versions of Research Unix (v8, v9, and v10) were similar failures earlier on, with almost nothing from them making it into mainstream Unix.

  (And some of the things that did make the transition were terribly mangled, cf Dennis Ritchie's famous remark that 'streams means something different when shouted'.)

• Unix vendors in general punted on improving Unix and actually evolving it to be more Unixy for most of the mid to late 1990s. Of course most of them were preoccupied with dying at the time, which didn't help much.

• Once the era of free Unixes got rolling the open source *BSDs clung desperately to past history, some of which was better off replaced, while Linux reinvented everything, not always very well. Everyone mostly ignored the lessons of Plan 9, late Research Unix, and often even past Unix experience, despite having a golden opportunity to incorporate it into their systems.

And this is a vastly incomplete list of such disappointments.

If you started out as a purist, the past 25+ years of Unix history have mostly been a series of disappointments (some of them fairly crushing). Pick a thing that you cared about and you've probably had your dreams and hopes dashed. I think it's really hard to stay a purist when you're let down so many times and you have to work with such a parade of increasingly disappointing stuff. You can burn out so much that you leave Unix entirely, or you can move slowly from being a passionate purist to being a resigned pragmatist.

(There are passionate purists who have stuck to their guns and still work in Unix, but I don't think that there are all that many of them. You need a lot of fortitude, determination, and sheer bloody-mindedness.)

As a result, I kind of envy and admire the recent Unix people who are passionate purists right now. From my position on the sidelines in an easy chair, I want to cheer them on and tell them 'cling to your idealism as long as possible before the world grinds you down with disappointments'. And who knows? Maybe the next decade or two of Unix will be great, full of technical wins for the passionate. I'd certainly be happy to see Unix beauty win for once.

Sidebar: When I realized that I was becoming a pragmatist

As it happens, I can fairly precisely pinpoint when I became aware that I was turning into a pragmatist. It was when I had the opportunity to run Plan 9 instead of Unix but decided to pass because it would be too much of a hassle, both because of the restrictive hardware needs and due to how fairly cut off from the Unix computing environment around me I'd be. I looked at just how many things I'd be going without and said 'nope, the pure appeal and beauty of Plan 9 is not enough to get me to do this'.

(There's a part of me that's always regretted that I took the easy way.)

Why people were enthused about gcc early on in its life

Rob Landley recently wrote a widely linked piece on the BSD/System V split (via). There are any number of parts of his piece that I disagree with, but today I want to talk specifically about the early history of gcc, or specifically why people were enthused about it in the late 1980s. The origin of gcc is pretty straightforward; Richard Stallman knew he needed a free C compiler to enable his goal of a completely free Unix, so he wrote one fairly early in the FSF's life. But that's not the reason most people became enthused about it after it got released; after all, at that point pretty much everyone already had a C compiler.

What they didn't have was a good compiler. As hard as it may be to believe now (in a world where everyone knows that a good part of your benchmark performance comes from the compiler), back in the late 1980s the CISC-based Unix vendors didn't really see their C compiler as a competitive advantage. By and large most people simply ported or used the basic Portable C Compiler that had originally come from Research Unix, maybe added some optimizations, and then called it a day. If it could compile Unix correctly and didn't run terribly badly, it was pretty much good enough for DEC (for Ultrix), Sun, SGI, HP, and so on. Devoting more than a bit of engineering resources to the C compiler was apparently too much of a hard sell. And of course if you were running real BSD Unix on your Vax you didn't even have the benefit of any compiler improvements DEC had made.

(This is less crazy than it may seem today. At the time probably the major locus of performance oriented computing was in Fortran code; the vendors did tend to have good Fortran compilers, which they generally wanted extra money for. And remember that C was a relatively new language in the 1980s.)

Then along came gcc. Gcc was rather more modern than common Unix C compilers and even in its early state it often generated clearly better (ie faster) code on Vaxes and M68K machines. In the academic environments I hung around in, it soon became common knowledge (or at least folklore) that recompiling programs like the X server with gcc would get you visible performance improvements. Since the machines back then were not exactly fast, 'CC=gcc' started appearing in more and more Makefiles, configuration systems started preferring it when it was found, and so on. That you could basically get a performance improvement for free just by installing gcc made all sorts of people quite interested and enthused about it.

(The other attractive thing about gcc for garden variety people was that it often supported a more modern dialect of C than the normal Unix compiler. On top of that, a newer and thus better version of gcc was often just a ftp fetch and rebuild away; you didn't have to wait for an entire OS update.)

This gcc advantage was only an advantage on the common CISC architectures. RISC based Unix systems had much better compilers from the get go and if I remember right gcc version 1 actually had architectural limitations that meant it couldn't really do the advanced optimizations that vendor compilers did. Thus the irony of people switching to gcc when Sun yanked the free compilers from later versions of their OS was that they were going backwards in performance. And yes, we knew this full well. But free beat non-free for many people, especially at universities with limited budgets.

(People who used MIPS based systems from MIPS and then SGI were in luck, because I believe that SGI basically always bundled the generally excellent MIPSco compilers (perhaps only for universities). We didn't use gcc much on either our MIPS Ultrix machines or on the SGI machines that succeeded them.)

Update: There's more good stuff in this Hacker News comment and comment thread.