Wandering Thoughts archives

Finding free numbers in a range, crudely, with Unix tools

Suppose, not entirely hypothetically, that you have the zone file for the reverse mapping of a single /24 subnet (in Bind format) and you want to find a free IP address in that subnet. The first field in the zone file is the last octet of the IP (ie for '127.0.0.1' it is '1'), so this problem reduces to finding what numbers are not used in the file out of the range 1 to 254.

So let's do this really crudely with readily available Unix tools:

grep '^[0-9]' ptrzone | awk '{print $1}' | sort >/tmp/used
seq 1 254 | sort >/tmp/all
comm -13 /tmp/used /tmp/all | sort -n

(We must do a 'sort -n' only at the end because of a stupid GNU comm decision.)

We can get rid of one of the two files here by piping directly into comm, but unfortunately we can't get rid of both of them in standard Bourne shell. In Bash and zsh (and AT&T ksh), we can use what they each call 'process substitution' to directly embed each pipeline into the comm arguments:

comm -13 <(grep '^[0-9]' ptrzone | awk '{print $1}' | sort) <(seq 1 254 | sort) | sort -n

(This also works in rc, with a different syntax.)

Once upon a time I would have happily written the single-line version. These days my tendencies are towards the multi-line version unless I'm feeling excessively clever, partly because it's easier to check the intermediate steps.

(A neater version would condense runs of numbers into ranges, but I don't think you can do that simply with any existing common Unix tools. Perl experts can maybe do it in a single line.)

PS: One hazard of a very well stuffed $HOME/bin directory is that it turns out I already had a command to do this for me (and with the neater range-based output). Oh well, such things happen. Perhaps I should thin out my $HOME/bin on the grounds of disuse of much of it, but it's always hard to get rid of your personal little scripts.

NFS hard mounts versus soft mounts

On most Unix systems NFS mounts come in your choice of two flavours, hard or soft. The Linux nfs manpage actually has a very good description of the difference; the short summary is that a hard NFS mount will keep trying NFS operations endlessly until the server responds while a soft NFS mount will give up and return errors after a while.

You can find people with very divergent opinions about which is better (cf, 2). My opinion is fairly strongly negative about soft mounts. The problem is that it is routine for a loaded NFS server to not respond to client requests within the client timeout interval because the timeout is not for the NFS server to receive the request, it's for the server to fully process it. As you might imagine, a server under heavy IO and network load may not be able to finish your disk IO for some time, especially if it's write IO. This makes NFS timeouts that would trigger soft NFS mount errors a relatively routine event in many real world environments.

(On Linux, any time a client reports 'nfs: server X not responding, still trying' that would be an IO error on a soft NFS mount. In our fileserver environment, some of these happen nearly every day.)

Many Unix programs do not really expect their IO to fail. Even programs that do notice IO errors often don't and can't do anything more than print an error message and perhaps abort. This is not a helpful response to transient errors, but then Unix programs are generally not really designed for a world with routine transient IO errors. Even when programs report the situation, users may not notice or may not be prepared to do very much except, perhaps, retry the operation.

(Write errors are especially dangerous because they can easily cause you to permanently lose data, but even read errors will cause you plenty of heartburn.)

Soft NFS mounts primarily make sense when you have some system that absolutely must remain responsive and cannot delay for too long for any reason. In this case a random but potentially very long kernel imposed delay is a really bad thing and you'd rather have the operation error out entirely so that your user level code can take action and at least respond in some way. Some NFS clients (or just specific NFS mounts) are only used in this way, for a custom system, and are not exposed to general use and general users.

(IO to NFS hard mounts can still be interrupted if you've sensibly mounted them with the intr option. It just requires an explicit decision at user level that the operation should be aborted, instead of the kernel deciding that all operations that have taken 'too long' should be aborted.)

PS: My bias here is that I've always been involved in running general use NFS clients, ones where random people will be using the NFS mounts for random and varied things with random and varied programs of very varied quality. This is basically a worst case for NFS soft mounts.

(Probably) Why Bash imports functions from the environment

In the wake of the Shellshock issues, a lot of people started asking why Bash even had a feature to import functions from the environment. The obvious answer is to allow subshell Bashes to inherit functions from parent shells. Now, you can come up with some clever uses for this feature (eg to pass very complex options down from parents to children), but as it happens I have my own views about why this feature probably originally came to exist.

Let us rewind to a time very long ago, like 1989 (when this feature was introduced). In 1989, Unix computers were slow. Very slow. They were slow to read files, especially if you might be reading your files over the network from a congested NFS server, and they were slow to parse and process files once they were loaded. This was the era in which shells were importing more and more commands as builtins, because not having to load and execute programs for things like test could significantly speed up your life. A similar logic drove the use of shell functions instead of shell scripts; shell functions were already resident and didn't require the overhead of starting a new shell and so on and so forth.

So there you are, with your environment all set up in Bash and you want to start an interactive subshell (from inside your editor, as a new screen window, starting a new xterm, or any number of other ways). Bash supports a per-shell startup file in .bashrc, so you could define all your shell functions in it and be done. But if you did this, your new subshell would have to open and read and parse and process your .bashrc. Slowly. In fact every new subshell would have to do this and on a slow system the idea of cutting out almost all of this overhead is very attractive (doing so really will make your new subshell start faster).

Bash already exports and imports plain environment variables, but those aren't all you might define in your .bashrc; you might also define shell functions. If a subshell could be passed shell functions from the environment, you could bypass that expensive read of .bashrc by pre-setting the entire environment in your initial shell and then just having them inherit it all. On small, congested 1989 era hardware (and even for years afterwards) you could get a nice speed boost here.

(This speed boost was especially important because Bash was already a fairly big and thus slow shell by 1989 standards.)

By the way, importing shell functions from the environment on startup is such a good idea that it was implemented at least twice; once in Bash and once in Tom Duff's rc shell for Plan 9.

(I don't know for sure which one was first but I suspect it was rc.)