Wandering Thoughts archives

Unix has been bad before

These days it's popular to complain about the terrible state of software on modern Linux machines, with their tangle of opaque DBus services, weird Gnome (or KDE) software, and the requirement for all sorts of undocumented daemons to do anything. I've written a fair amount of entries like this myself. But make no mistake, Linux is not uniquely bad here and is not some terrible descent from a previous state of Unix desktop grace.

As I've alluded to before, the reality is that all of the old time Unix workstation vendors did all sorts of similarly terrible things themselves, back in the days when they were ongoing forces. No Unix desktop has ever been a neat and beautiful thing under the hood; all of them have been ugly and generally opaque conglomerations of wacky ideas. Sometimes these ideas spilled over into broader 'server' software and caused the expected heartburn in sysadmins there.

To the extent that the Unixes of the past were less terrible than the present, my view is that this is largely because old time Unix vendors were constrained by more limited hardware and software environments. Given modern RAM, CPUs, and graphics hardware and current software capabilities, they probably would have done things that are at least as bad as Linux systems are doing today. Instead, having only limited RAM and CPU power necessarily limited their ability to do really bad things (at least usually).

(One of the reasons that modern Linux stuff is better than it could otherwise be is that at least some of the people creating it have learned from the past and are thereby avoiding at least some of the mistakes people have already made.)

Also, while most of the terrible things have been confined to desktop Unix, not all of them were. Server Unix has seen its own share of past bad mistakes from various Unix vendors. Fortunately they tended to be smaller mistakes, if only because a lot of vendor effort was poured into desktops (well, most of the time; let's not talk about how the initial SunOS 4 releases ran on servers).

The large scale lesson I take from all of this is that Unix (as a whole) can and will recover from things that turn out to be mistakes. Sometimes it's a rocky road that's no fun during things, but we get there eventually.

The pending delete problem for Unix filesystems

Unix has a number of somewhat annoying filesystem semantics that tend to irritate designers and implementors of filesystems. One of the famous ones is that you can delete a file without losing access to it. On at least some OSes, if your program open()s a file and then tries to delete it, either the deletion fails with 'file is in use' or you immediately lose access to the file; further attempts to read or write it will fail with some error. On Unix your program retains access to the deleted file and can even pass this access to other processes in various ways. Only when the last process using the file closes it will the file actually get deleted.

This 'use after deletion' presents Unix and filesystem designers with the problem of how you keep track of this in the kernel. The historical and generic kernel approach is to keep both a link count and a reference count for each active inode; an inode is only marked as unused and the filesystem told to free its space when both counts go to zero. Deleting a file via unlink() just lowers the link count (and removes a directory entry); closing open file descriptors is what lowers the reference count. This historical approach ignored the possibility of the system crashing while an inode had become unreachable through the filesystem and was only being kept alive by its reference count; if this happened the inode became a zombie, marked as active on disk but not referred to by anything. To fix it you had to run a filesystem checker, which would find such no-link inodes and actually deallocate them.

(When Sun introduced NFS they were forced to deviate slightly from this model, but that's an explanation for another time.)

Obviously this is not suitable for any sort of journaling or 'always consistent' filesystem that wants to avoid the need for a fsck after unclean shutdowns. All such filesystems must keep track of such 'deleted but not deallocated' files on disk using some mechanism (and the kernel has to support telling filesystems about such inodes). When the filesystem is unmounted in an orderly way, these deleted files will probably get deallocated. If the system crashes, part of bringing the filesystem up on boot will be to apply all of the pending deallocations.

Some filesystems will do this as part of their regular journal; you journal, say, 'file has gone to 0 reference count', and then you know to do the deallocation on journal replay. Some filesystems may record this information separately, especially if they have some sort of 'delayed asynchronous deallocation' support for file deletions in general.

(Asynchronous deallocation is popular because it means your process can unlink() a big file without having to stall while the kernel frantically runs around finding all of the file's data blocks and then marking them all as free. Given that finding out what a file's data blocks are often requires reading things from disk, such deallocations can be relatively slow under disk IO load (even if you don't have other issues there).)

PS: It follows that a failure to correctly record pending deallocations or properly replay them is one way to quietly lose disk space on such a journaling filesystem. Spotting and fixing this is one of the things that you need a filesystem consistency checker for (whether it's a separate program or embedded into the filesystem itself).

Unix's pipeline problem (okay, its problem with file redirection too)

In a comment on yesterday's entry, Mihai Cilidariu sensibly suggested that I not add timestamp support to my tools but instead outsource this to a separate program in a pipeline. In the process I would get general support for this and complete flexibility in the timestamp format. This is clearly and definitely the right Unix way to do this.

Unfortunately it's not a good way in practice, because of a fundamental pragmatic problem Unix has with pipelines. This is our old friend block buffering versus line buffering. A long time ago, Unix decided that many commands should change their behavior in the name of efficiency; if they wrote lines of output to a terminal you'd get each line as it was written, but if they wrote lines to anything else you'd only get it in blocks.

This is a big problem here because obviously a pipeline like 'monitor | timestamp' basically requires the monitor process to produce output a line at time in order to be useful; otherwise you'd get large blocks of lines that all had the same timestamp because they were written to the timestamp process in a block. The sudden conversion from line buffered to block buffered can also affect other sorts of pipeline usage.

It's certainly possible to create programs that don't have this problem, ones that always write a line at a time (or explicitly flush after every block of lines in a single report). But it is not the default, which means that if you write a program without thinking about it or being aware of the issue at all you wind up with a program that has this problem. In turn people like me can't assume that a random program we want to add timestamps to will do the right thing in a pipeline (or keep doing it).

(Sometimes the buffering can be an accidental property of how a program was implemented. If you first write a simple shell script that runs external commands and then rewrite it as a much better and more efficient Perl script, well, you've probably just added block buffering without realizing it.)

In the end, what all of this really does is that it chips away quietly at the Unix ideal that you can do everything with pipelines and that pipelining is the right way to do lots of stuff. Instead pipelining becomes mostly something that you do for bulk processing. If you use pipelines outside of bulk processing, sometimes it works, sometimes you need to remember odd workarounds so that it's mostly okay, and sometimes it doesn't do what you want at all. And unless you know Unix programming, why things are failing is pretty opaque (which doesn't encourage you to try doing things via pipelines).

(This is equally a potential problem with redirecting program output to files, but it usually hits most acutely with pipelines.)

What I want to have in shell (filename) completion

I've been using basic filename completion in my shell for a while now, and doing so has given me a perspective on what advanced features of this I'd find useful and which strike me as less useful. Unfortunately for me, the features that I'd find most useful are the ones that are the hardest to implement.

Put simply, the problem with basic filename completion is that any time you want to use even basic shell features like environment variables, you lose completion. Do you refer to some directories through convenience variables? Nope, not any more, because you can choose between completing a long name and not completing, say, '$ml/afilename'.

(I know, bash supports completing filenames that use environment variables. Probably zsh does as well. I'm not interested in switching to either right now.)

But environment variables are just one of the ways to shorten filenames. Two more cases are using wildcards to match unique or relatively unique subsets of a long filename and using various multi-match operators to specify several filenames in some directory or the like. Both of these would be handy to be able to do filename completion for. In fact, let's generalize that: what I'd really like is for my shell to be able to do filename completion in the face of any and all things that can appear in filenames and get expanded by the shell. Then I could combine the power of filename completion and the power of all of those handy shell operators together.

Of course this is easier said than done. I know that what I'm asking for is quite a challenging programming exercise and to some extent a design exercise once we get to the obscure features. But it sure would be handy (more handy, in my biased opinion, than a number of other completion features).

(I've never used eg bash's smart context aware autocompletion of program arguments in general, so I don't know for sure how useful I'd find it; however, my personal guess is 'not as much as full filename completion'. I'm so-so on autocompletion of program names; I suppose I do sometimes use programs with annoyingly long names, so my shell might as well support it. Again I'd rate it well below even autocompletion with environment variables, much less the full version.)