Wandering Thoughts archives

The (Unix) shell is not just for running programs

In the Reddit comments on yesterday's entry, I ran across the following comment:

No. The shell literally has the sole purpose of running external programs. Anything more is extra.

The V1 shell read a line, split on whitespace, and executed the command from /bin. You could change the current directory from in the shell, that was it.

On any version of Unix as far back as at least V7, this is false. The Unix shell may have started out simply being a way to run programs, but it long ago stopped being just that. Since the V7 shell is a ground up rewrite, one cannot even argue that the shell simply drifted into these additional features for convenience. The V7 shell was consciously designed from scratch, and as part of that design it included major programming features including control flow constructs drawn directly from the general Algol line of computer language design. Inclusion of these programming features is not an accident and not a drift over time; it is a core part of the shell's design and thus its intended purpose. The V7 shell is there both to run programs and to write programs (shell scripts), and this is completely intended.

(In terms of control flow, I'm thinking here of if, while, and for, and there's also case.)

In short, the shell as in part a programming language is part of Unix's nature from at least the first really popular Unix version (V7 became the base of many further lines of Unix). To the extent that the Unix design ethos or philosophy exists as a coherent thing, it demonstrably includes a strongly programmable shell.

You can make an argument that the V6 shell (the 'Mashey shell') shows this too, but it was apparently a derivative of and deliberately backwards compatible with the original 'just run things' Thompson shell. The V7 Bourne shell is a clear, from scratch break with the original Thompson shell, and it was demonstrably accepted by Research Unix as being, well, proper Unix.

(If you want even more proof that Research Unix's view of the shell includes programming, the shell was reimplemented once again for Version 10 and Plan 9 in the form of Tom Duff's rc shell and, you guessed it, that included programmability too, this time with more C-like syntax instead of the Algol-like syntax of the Bourne shell.)

(You can argue that this conjoining of 'just run programs for people' and 'write shell scripts' in a single program is a mistake and these roles should be split apart into two programs, but that's a different argument. I happen to think that it's also wrong, and on more than one level.)

One thing that makes the Bourne shell an odd language

In many ways, the Bourne shell is a relatively conventional programming language. It has a few syntactic abnormalities, a few flourishes created by the fact that it is an engine for running programs (although other languages have featured equivalents of $(...) in the form of various levels of 'eval' functionality), and a different treatment of unquoted words, but the overall control structure is an extremely familiar Algol-style one (which is not surprising, since Steve Bourne really liked Algol).

But the Bourne shell does have one thing that clearly makes it an odd language, namely that it has outsourced what are normally core language functions to external programs. Or rather it started out in its original version by outsourcing those functions; versions of the Bourne shell since then have pulled them back in in various ways. Here I am thinking of both evaluating conditionals via test aka [ and arithmetic via expr (which also does some other things too).

(Bourne shells have had test as a builtin for some time (sometimes with some annoyances) and built in arithmetic is often present these days as $...)

There's no reason why test has to be a separate program and neither test nor expr seems to have existed in Research Unix V6, so they both appeared in V7 along with the Bourne shell itself. They aren't written in BourneGol, so they may not have been written by Steve Bourne himself, but at least test was clearly written as a companion program (the V7 Bourne shell manpage explicitly mentions it, among other things).

I don't know why the original Bourne shell made this decision. It's possible that it was simply forced by the limitations of the PDP-11 environment of V7. Maybe a version of the Bourne shell with test and/or expr built into the main shell code would have either been too big or just considered over-bloated for something that would mostly be used interactively (and thus not be using test et al very often). Or possibly they were just easier to write as separate programs (the V7 expr is just a single yacc file).

Note that there are structural reasons in the Bourne shell to make if et al conditions be the result of commands, instead of restricting them to (only) be actual conditions. But the original Bourne shell could have done this with test or the equivalent as a built-in command, and it certainly has other built in commands. Perhaps test needing to be an actual command was one of the things that pushed it towards not being built in. You can certainly see a spirit of minimalism at work here if you want to (although I have no idea if that's the reason).

(This expands on a tweet of mine.)

Sidebar: It's not clear when test picked up its [ alias

Before I started writing this entry, I expected that test was also known as [ right from the beginning in V7. Now I'm not so sure. On the one hand, the actual V7 shell scripts I can find eg here consistently use test instead of [ and the V7 compile scripts don't seem to create a [ hardlink. On the other hand, the V7 test source already has special magic handling if it's invoked as [.

(There are V7 disk images out there that you can boot up on a PDP-11 emulator, so in theory I could fire one up and see if it has a /bin/[. In practice I'm not that energetic.)