Wandering Thoughts archives

The practical result of OpenBSD's support policy

Recently I read Ted Unangst's long term support considered harmful (via), where he mentions OpenBSD's relatively short term support policy (it's one year at most and less in practice) and then says:

Now on the one hand, this forces users to upgrade at least once per year. On the other hand, this forces users to upgrade at least once per year. [...]

Oh, how this makes me laugh. Sadly.

We have a not insignificant number of OpenBSD firewalls. I don't believe that any of them are running a currently supported OpenBSD release and if they are it's not because we upgraded them, it's because they were set up recently. Other than that we leave the firewalls strictly alone until we have to change them for some reason.

(One of the reasons we never attempt to upgrade firewalls is that OpenBSD more or less explicitly doesn't have backwards compatibility between releases in things like PF; OpenBSD can and has changed PF syntax, rules, and rule handling around from release to release. When the core of your firewall may have changed, upgrades are not a 'cvs up; recompile', they are a full exercise of 'install a spare machine then retest and requalify everything from scratch' (which in our small environment is done by hand on scrounged hardware). Deployment of the result has its own pains.)

I don't think we're alone here; I suspect that there are lots of people running OpenBSD releases that are out of support. OpenBSD's short support period certainly accomplishes the goal of less (valid) bug reports to OpenBSD and less work for OpenBSD to do, but it doesn't necessarily either get users to upgrade or reduce the actual bugs that they may encounter. Instead the effect of this short support period and lack of long term support is to maroon more OpenBSD users without any support at all.

I doubt that OpenBSD cares about such marooned users, but as usual I feel like mentioning the practical results of a policy, not just the theoretical ones.

All of this leads me to laugh very hollowly at Ted Unangst's conclusion that:

A one year support window isn't too short; it's too long.

I'm pretty certain that the major goal this would achieve would be to allow OpenBSD to reject even more bug reports.

(There is a general lesson here but I'm going to leave it to another entry.)

Sidebar: OpenBSD and practical support

To put it very simply, we aren't worried much by the lack of support we have with our current firewalls because we don't expect any support and I don't think we'll ever file any bug reports with OpenBSD even if we find issues (which we do from time to time). Our attitude on the state of OpenBSD is 'we get what we get and it's nice if it works'. If it doesn't work, we find something that does.

(This is why we ran Fedora L2TP VPN servers for a while.)

Some people might consider this lack of bug reports to be antisocial, but I have some opinions for them.

What /etc/shells is and isn't

In traditional Unix, /etc/shells has only one true purpose: it lists programs that chsh will let you change your shell to (if it lets you do anything). Before people are tempted to make other programs use this file for something else, it is important to understand the limits of /etc/shells. These include but are not limited to:

• Logins may have /etc/passwd entries that list other shells. For example, back when restricted shells were popular it was extremely common to not list them in /etc/shells so you couldn't accidentally chsh yourself into a restricted shell and then get stuck.

  Some but not all programs have used the absence of a shell from /etc/shells as a sign that it is a restricted shell (or not a real shell at all) and they should restrict a user with that shell in some way. Other programs have used different tests, such as matching against specific shell names or name prefixes.

  (It's traditional for the FTP daemon to refuse access for accounts that do not have a shell that's in /etc/shells and so this is broadly accepted. Other programs are on much thinner ice.)

• On the other hand, sometimes you can find restricted shells in /etc/shells; a number of systems (Ubuntu and several FreeBSD versions) include rbash, the restricted version of Bash, if it's installed.

• Not all normal shells used in /etc/passwd or simply installed on the system necessarily appear in /etc/shells for various reasons. In practice there are all sorts of ways for installed shells to fall through the cracks. Of course this makes them hard to use as your login shell (since you can't chsh to them), but this can be worked around in various ways.

  For example, our Ubuntu systems have /bin/csh and /bin/ksh (and some people use them as their login shells) but neither are in /etc/shells.

• The (normal and unrestricted) shell someone's actually using isn't necessarily in either /etc/shells or their /etc/passwd entry. Unix is flexible and easily lets you use $SHELL and some dotfile hacking to switch basically everything over to running your own personal choice of shell, per my entry on using an alternate shell.

  (Essentially everything on Unix that spawns what is supposed to be your interactive shell has been clubbed into using $SHELL, partly because the code to use $SHELL is easier to write than the code to look up someone's /etc/passwd entry to find their official login shell. This feature probably came into Unix with BSD Unix, which was basically the first Unix to have two shells.)

• Entries in /etc/shells don't necessarily exist.
• Entries in /etc/shells are not necessarily shells. Ubuntu 14.04 lists screen.

• Not all systems even have an /etc/shells. Solaris and derivatives such as Illumos and OmniOS don't.

In the face of all of this, most programs should simply use $SHELL and assume that it is what the user wants and/or what the sysadmin wants the user to get. It's essentially safe to assume that $SHELL always exists, because it is part of the long-standing standard Unix login environment. As a corollary, a program should not change $SHELL unless it has an excellent reason to do so.

Note particularly that a user's $SHELL not being listed in /etc/shells means essentially nothing. As outlined above, there are any number of non-theoretical ways that this can and does happen on real systems that are out there in the field. As a corollary your program should not do anything special in this case unless it has a really strong reason to do so, generally a security-related reason. Really, you don't even want to look at /etc/shells unless you're chsh or ftpd or sudo or the like.

(This entry is sadly brought to you by a program getting this wrong.)

ZFS should be your choice today if you need an advanced filesystem on Unix

The other day I ran down why ZFS is your only real choice on Linux if you need an advanced filesystem. Well, it's not just Linux. I don't know enough about the landscape of filesystems on other Unixes to confidently say it's your only choice on non-Linux Unixes, but I do think it is by far your best choice for this job.

Let's start with the first and biggest reason: ZFS works and has for years. At this point ZFS has been running in production environments (some of them very big and/or demanding) for more than half a decade. Yes, development is still happening and people are still finding issues (just as they are with essentially any filesystem), but the bulk of ZFS (in code and features) is extensively battle tested and is highly unlikely to blow up on you. Using ZFS puts you nowhere near the bleeding edge, and I don't think there's any other advanced Unix filesystem that can really make that claim.

The second reason is that ZFS is cross-platform, and in fact with care and advanced planning you can move ZFS pools between Unixes (instead of just shifting from one Unix to another for your next storage server build or the like). Chosing ZFS does not lock you into Solaris the way it once did; these days you can run ZFS reasonably sensibly on your choice of Oracle Solaris, anything Illumos based (eg OmniOS or SmartOS), FreeBSD, and Linux. People are running ZFS in production on all of these. This covers almost all of the gamut of major remaining Unixes and gives you plenty of choices as far as style, administration, user level tools, and so on. And if your first choice of Unix for ZFS turns out to have problems, well, no big deal; you can move your basic architecture to another Unix. In a way you can treat ZFS much the way you treat applications like Apache or Exim and basically host it on the Unix substrate of your choice.

(Yes, yes, the OpenBSD and NetBSD and DragonflyBSD and OS X Server crowds are now annoyed at my over-generalization. Sorry, you're all relatively marginal in this space at this time.)

Finally, and it's high time I said this explicitly, ZFS not merely works but it's decently good and has generally made the right design decisions. I can quibble with some of its architectural choices and it's (still) missing a few features that people continually yearn for, but on the whole it is not merely a filesystem that is there but a filesystem that is actually nice, with choices that are sensible and usable.

This last reason is actually kind of important. If there are no other production ready advanced filesystems on Unixes (Linux or otherwise) then ZFS would win today by default. But if it kind of sucked, you might want to stick with current filesystems and wait unless data integrity was really, really important to you. As it is ZFS is decent and deciding to use ZFS doesn't constrain your Unix choices much, so I think you should seriously consider using ZFS for its data integrity advantages.

(There are any number of situations where using ZFS is still not worth the hassles or the risks. Or to be direct, ZFS is certainly not to the point on Linux and maybe FreeBSD where I think using it is a slam dunk obvious choice. Of course on Solaris or Illumos you basically have no option, so go with the flow.)

Does having a separate daemon manager help system resilience?

One of the reasons usually put forward for having a separate daemon manager process (instead of having PID 1 do this work) is that doing so increases overall system resilience. As the theory goes, PID 1 can be made minimal and extremely unlikely to crash (unlike a more complex PID 1), while if the more complicated daemon manager does crash it can be restarted.

Well, maybe. The problem is the question of how well you can actually take over from a crashed daemon manager. Usually this won't be an orderly takeover and you can't necessarily trust anything in any auxiliary database that the daemon manager has left behind (since it could well have been corrupted before or during the crash). You need to have the new manager process step in and somehow figure out what was (and is) running and what isn't, then synchronize the state of the system back to what it's supposed to be, then pick up monitoring everything.

The simple case is a passive init system. Since the init system does not explicitly track daemon state, there is no state to recover on a daemon manager restart and resynchronization can be done simply by trying to start everything that should be started (based on runlevel and so on). We can blithely assume that the 'start' action for everything will do nothing if the particular service is already started. Of course this is not very realistic, as passive init systems generally don't have daemon manager processes that can crash in the first place.

For an active daemon manager, I think that at a minimum what you need is some sort of persistent and stable identifier for groups of processes that can be introspected and monitored from an arbitrary process. The daemon manager starts processes for all services under a an identifier determined from their service name; then when it crashes and you have to start a new one, the new one can introspect the identifiers for all of the groups to determine what services are (probably) running. Unfortunately there are lots of complications here, including that this doesn't capture the state of 'one-shot' services without persistent processes. This is of course not a standard Unix facility, so no fully portable daemon manager can do this.

It's certainly the case that a straightforward, simple daemon manager will not be able to take over from a crashed instance of itself. Being able to do real takeover requires both system-specific features and a relatively complex design and series of steps on startup, and still leaves you with uncertain or open issues. In short, having a separate daemon manager does not automatically make the system any more resilient under real circumstances. A crashing daemon manager is likely to force a system reboot just as much as a crashing PID 1 does.

However I think it's fair to say that under normal circumstances a separate daemon manager process crashing (instead of PID 1 crashing) will buy you more time to schedule a system outage. If the only thing that needs the daemon manager running is starting or stopping services and you already have all normal services started up, your system may be able to run for days before you need to reboot it. If your daemon manager is more involved in system operation or is routinely required to restart services, well, you're going to have (much) less time depending on the exact details.

How init wound up as Unix's daemon manager

If you think about it, it's at least a little bit odd that PID 1 wound up as the de facto daemon manager for Unix. While I believe that the role itself is part of the init system as a whole, this is not the same thing as having PID 1 do the job and in many ways you'd kind of expect it to be done in another process. As with many things about Unix, I think that this can be attributed to the historical evolution Unix has gone through.

As I see the evolution of this, things start in V7 Unix (or maybe earlier) when Research Unix grew some system daemons, things like crond. Something had to start these, so V7 had init run /etc/rc on boot as the minimal approach. Adding networking to Unix in BSD Unix increased the number of daemons to start (and was one of several changes that complicated the whole startup process a lot). Sun added even more daemons with NFS and YP and so on and either created or elaborated interdependencies among them. Finally System V came along and made everything systematic with rcN.d and so on, which was just in time for yet more daemons.

(Modern developments have extended this even further to actively monitoring and restarting daemons if you ask them to. System V init could technically do this if you wanted, but people generally didn't use inittab for this.)

At no point in this process was it obvious to anyone that Unix was going through a major sea change. It's not as if Unix went in one step from no daemons to a whole bunch of daemons; instead there was a slow but steady growth in both the number of daemons and the complexity of system startup in general, and much of this happened on relatively resource-constrained machines where extra processes were a bad idea. Had there been a single giant step, maybe people would have sat down and asked themselves if PID 1 and a pile of shell scripts were the right approach and said 'no, it should be a separate process'. But that moment never happened; instead Unix basically drifted into the current situation.

(Technically speaking you can argue that System V init actually does do daemon 'management' in another process. System V init doesn't directly start daemons; instead they're started several layers of shell scripts away from PID 1. I call it part of PID 1 because there is no separate process that really has this responsibility, unlike the situation in eg Solaris SMF.)

Finding free numbers in a range, crudely, with Unix tools

Suppose, not entirely hypothetically, that you have the zone file for the reverse mapping of a single /24 subnet (in Bind format) and you want to find a free IP address in that subnet. The first field in the zone file is the last octet of the IP (ie for '127.0.0.1' it is '1'), so this problem reduces to finding what numbers are not used in the file out of the range 1 to 254.

So let's do this really crudely with readily available Unix tools:

grep '^[0-9]' ptrzone | awk '{print $1}' | sort >/tmp/used
seq 1 254 | sort >/tmp/all
comm -13 /tmp/used /tmp/all | sort -n

(We must do a 'sort -n' only at the end because of a stupid GNU comm decision.)

We can get rid of one of the two files here by piping directly into comm, but unfortunately we can't get rid of both of them in standard Bourne shell. In Bash and zsh (and AT&T ksh), we can use what they each call 'process substitution' to directly embed each pipeline into the comm arguments:

comm -13 <(grep '^[0-9]' ptrzone | awk '{print $1}' | sort) <(seq 1 254 | sort) | sort -n

(This also works in rc, with a different syntax.)

Once upon a time I would have happily written the single-line version. These days my tendencies are towards the multi-line version unless I'm feeling excessively clever, partly because it's easier to check the intermediate steps.

(A neater version would condense runs of numbers into ranges, but I don't think you can do that simply with any existing common Unix tools. Perl experts can maybe do it in a single line.)

PS: One hazard of a very well stuffed $HOME/bin directory is that it turns out I already had a command to do this for me (and with the neater range-based output). Oh well, such things happen. Perhaps I should thin out my $HOME/bin on the grounds of disuse of much of it, but it's always hard to get rid of your personal little scripts.

NFS hard mounts versus soft mounts

On most Unix systems NFS mounts come in your choice of two flavours, hard or soft. The Linux nfs manpage actually has a very good description of the difference; the short summary is that a hard NFS mount will keep trying NFS operations endlessly until the server responds while a soft NFS mount will give up and return errors after a while.

You can find people with very divergent opinions about which is better (cf, 2). My opinion is fairly strongly negative about soft mounts. The problem is that it is routine for a loaded NFS server to not respond to client requests within the client timeout interval because the timeout is not for the NFS server to receive the request, it's for the server to fully process it. As you might imagine, a server under heavy IO and network load may not be able to finish your disk IO for some time, especially if it's write IO. This makes NFS timeouts that would trigger soft NFS mount errors a relatively routine event in many real world environments.

(On Linux, any time a client reports 'nfs: server X not responding, still trying' that would be an IO error on a soft NFS mount. In our fileserver environment, some of these happen nearly every day.)

Many Unix programs do not really expect their IO to fail. Even programs that do notice IO errors often don't and can't do anything more than print an error message and perhaps abort. This is not a helpful response to transient errors, but then Unix programs are generally not really designed for a world with routine transient IO errors. Even when programs report the situation, users may not notice or may not be prepared to do very much except, perhaps, retry the operation.

(Write errors are especially dangerous because they can easily cause you to permanently lose data, but even read errors will cause you plenty of heartburn.)

Soft NFS mounts primarily make sense when you have some system that absolutely must remain responsive and cannot delay for too long for any reason. In this case a random but potentially very long kernel imposed delay is a really bad thing and you'd rather have the operation error out entirely so that your user level code can take action and at least respond in some way. Some NFS clients (or just specific NFS mounts) are only used in this way, for a custom system, and are not exposed to general use and general users.

(IO to NFS hard mounts can still be interrupted if you've sensibly mounted them with the intr option. It just requires an explicit decision at user level that the operation should be aborted, instead of the kernel deciding that all operations that have taken 'too long' should be aborted.)

PS: My bias here is that I've always been involved in running general use NFS clients, ones where random people will be using the NFS mounts for random and varied things with random and varied programs of very varied quality. This is basically a worst case for NFS soft mounts.

(Probably) Why Bash imports functions from the environment

In the wake of the Shellshock issues, a lot of people started asking why Bash even had a feature to import functions from the environment. The obvious answer is to allow subshell Bashes to inherit functions from parent shells. Now, you can come up with some clever uses for this feature (eg to pass very complex options down from parents to children), but as it happens I have my own views about why this feature probably originally came to exist.

Let us rewind to a time very long ago, like 1989 (when this feature was introduced). In 1989, Unix computers were slow. Very slow. They were slow to read files, especially if you might be reading your files over the network from a congested NFS server, and they were slow to parse and process files once they were loaded. This was the era in which shells were importing more and more commands as builtins, because not having to load and execute programs for things like test could significantly speed up your life. A similar logic drove the use of shell functions instead of shell scripts; shell functions were already resident and didn't require the overhead of starting a new shell and so on and so forth.

So there you are, with your environment all set up in Bash and you want to start an interactive subshell (from inside your editor, as a new screen window, starting a new xterm, or any number of other ways). Bash supports a per-shell startup file in .bashrc, so you could define all your shell functions in it and be done. But if you did this, your new subshell would have to open and read and parse and process your .bashrc. Slowly. In fact every new subshell would have to do this and on a slow system the idea of cutting out almost all of this overhead is very attractive (doing so really will make your new subshell start faster).

Bash already exports and imports plain environment variables, but those aren't all you might define in your .bashrc; you might also define shell functions. If a subshell could be passed shell functions from the environment, you could bypass that expensive read of .bashrc by pre-setting the entire environment in your initial shell and then just having them inherit it all. On small, congested 1989 era hardware (and even for years afterwards) you could get a nice speed boost here.

(This speed boost was especially important because Bash was already a fairly big and thus slow shell by 1989 standards.)

By the way, importing shell functions from the environment on startup is such a good idea that it was implemented at least twice; once in Bash and once in Tom Duff's rc shell for Plan 9.

(I don't know for sure which one was first but I suspect it was rc.)

Unnoticed nonportability in Bourne shell code (and elsewhere)

In response to my entry on how Bashisms in #!/bin/sh scripts aren't necessarily bugs, FiL wrote:

If you gonna use bashism in your script why don't you make it clear in the header specifying #!/bin/bash instead [of] #!/bin/sh? [...]

One of the historical hard problems for Unix portability is people writing non-portable code without realizing it, and Bourne shell code is no exception. This is true for even well intentioned people writing code that they want to be portable.

One problem, perhaps the root problem, is that very little you do on Unix will come with explicit (non-)portability warnings and you almost never have to go out of your way to use non-portable features. This makes it very hard to know whether or not you're actually writing portable code without trying to run it on multiple environments. The other problem is that it's often both hard to remember and hard to discover what is non-portable versus what is portable. Bourne shell programming is an especially good example of both issues (partly because Bourne shell scripts often use a lot of external commands), but there have been plenty of others in Unix's past (including 'all the world's a VAX' and all sorts of 64-bit portability issues in C code).

So one answer to FiL's question is that a lot of people are using bashisms in their scripts without realizing it, just as a lot of people have historically written non-portable Unix C code without intending to. They think they're writing portable Bourne shell scripts, but because their /bin/sh is Bash and nothing in Bash warns about things the issues sail right by. Then one day you wind up changing /bin/sh to be Dash and all sorts of bits of the world explode, sometimes in really obscure ways.

All of this sounds abstract, so let me give you two examples of accidentally Bashisms I've committed. The first and probably quite common one is using '==' instead of '=' in '[ ... ]' conditions. Many other languages use == as their string equality check, so at some point I slipped and started using it in 'Bourne' shell scripts. Nothing complained, everything worked, and I thought my shell scripts were fine.

The second I just discovered today. Bourne shell pattern matching allows character classes, using the usual '[...]' notation, and it even has negated characters classes. This means that you can write something like the following to see if an argument has any non-number characters in it:

case "$arg" in
   *[^0-9]*) echo contains non-number; exit 1;;
esac

Actually I lied in that code. Official POSIX Bourne shell doesn't negate character classes with the usual '^' character that Unix regular expressions use; instead it uses '!'. But Bash accepts '^' as well. So I wrote code that used '^', tested it, had it work, and again didn't realize that I was non-portable.

(Since having a '^' in your character class is not an error in a POSIX Bourne shell, the failure mode for this one is not a straightforward error.)

This is also a good example of how hard it is to test for non-portability, because even when you use 'set -o posix' Bash still accepts and matches this character class in its way (with '^' interpreted as class negation). The only way to test or find this non-portability is to run the script under a different shell entirely. In fact, the more theoretically POSIX compatible shells you test on the better.

(In theory you could try to have a perfect memory for what is POSIX compliant and not need any testing at all, or cross-check absolutely everything against POSIX and never make a mistake. In practice humans can't do that any more than they can write or check perfect code all the time.)

Why blocking writes are a good Unix API (on pipes and elsewhere)

One of the principles of good practical programming is that when your program can't make forward progress, it should do nothing rather than, say, continue to burn CPU while it waits for something to do. You want your program to do what work it can and then generally go to sleep, and thus you want APIs that encourage this to happen by default.

Now consider a chain of programs (or processes or services), each one feeding the next. In a multi-process environment like this you usually want something that gets called 'backpressure', where if any one component gets overloaded or can't make further progress it pushes back on the things feeding it so that they stop in turn (and so on back up the chain until everything quietly comes to a stop, not burning CPU and so on).

(You also want an equivalent for downstream services, where they process any input they get (if they can) but then stop doing anything if they stop getting any input at all.)

I don't think it's a coincidence that this describes classic Unix blocking IO to both pipes and files. Unix's blocking writes do backpressure pretty much exactly the way you want to happen; if any stage in a pipeline stalls for some reason, pretty soon all processes involved in it will block and sleep in write()s to their output pipe. Things like disk IO speed limits or slow processing or whatever will naturally do just what you want. And the Unix 'return what's available' behavior on reads does the same thing for the downstream of a stalled process; if the process wrote some output you can process it, but then you'll quietly go to sleep as you block for input.

And this is why I think that Unix having blocking pipe writes by default is not just a sensible API decision but a good one. This decision makes pipes just work right.

(Having short reads also makes the implementation of pipes simpler, because you don't have complex handling in the situation where eg process B is doing a read() of 128 megabytes while process A is trying to write() 64 megabytes to it. The kernel can make this work right, but it needs to go out of its way to do so.)