Wandering Thoughts archives

The somewhat apocryphal history of comments in the Bourne shell

(This is the story as I heard it and learned it in the past. It may or may not be actually accurate, but then a lot of Unix history is passed around as folklore, and in some ways the folklore is more important and more influential than the truth.)

In the beginning, in V7, the Bourne shell didn't actually have comments. Instead, all it had was :, which is not quite a comment (you can see this for yourself in the V7 sh manpage). As part of V7's minimalism, the kernel had no mechanism to execute shell scripts; instead, the shell did that itself when the kernel's exec() refused to run an executable file.

When people at Berkeley created csh they gave it actual comments, introduced by # characters. Having two shells created a dilemma, though; when csh went to run a shell script, how would it know which shell the script was written for? The solution was simple. Since the Bourne shell didn't have # comments, csh could look at the start of the file; if it had a #, it was a csh script, and otherwise it was a Bourne shell script.

(Hence you at least used to be able to find some very, very old Bourne shell scripts that started ': this is a sh script' or the like.)

Shortly after that the people at UCB came to their senses and invented '#!', which makes the kernel exec() mechanisms directly handle shell scripts. In order to make Bourne shell scripts compatible with this new mechanism, BSD added support for # comments to the Bourne shell. For backwards compatibility, they left support for the old 'executed directly by the shell when exec() fails' form of shell scripts in both csh and sh, complete with csh's special peeking for '#' and ':' (where it lingers on even today).

I am not sure when # made it into the System V version of the Bourne shell, although I think that it didn't take too long, possibly because it was seen as a generally sensible idea given :'s defects as a general comment mechanism.

(I believe that kernel support for '#!' took much longer and thus you couldn't use a shell script as a user login shell on System V machines for quite a while.)

Sidebar: an amusing experiment

Put the following in an executable file and try to run it from within various shells (with eg 'sh -c /tmp/exper') on various systems:

#
for i in 1; do
  echo bourne-like shell
done

Now, change the first line from '#' to ':' and try the experiment again.

(This is not the only example of Unix fossilization.)

The difference in the Bourne shell between : and #

I don't know how people learn the Bourne shell these days, but when I learned it, I first encountered ':' as a vaguely peculiar second way of writing comments, one used by some old shell scripts that hung around our systems. This is true as far as it goes, which is not necessarily very far.

The difference between # and : is that # starts a real comment that runs to the end of the line, while : is a real command that just does nothing (this is what makes the :; prompt trick work). As a real command, its 'arguments' are parsed and things in it can have potential side effects, which means that you have to be careful what you put after it. However, this also means that it can be used in places that require an actual command.

In the old days the classical example of this was in if statements, because the Bourne shell had not yet picked up a general negation operator. Instead you had to write:

if something ....; then
    : do nothing
else
    # do the interesting thing
fi

You couldn't use a # comment in place of the :, because the Bourne shell grammar rules require that there be a statement (and thus a command).

(Modern spec compliant Bourne shells have the general negation operator '!', so you can express this directly.)

There are obscure uses for : in other contexts; for example, if you want an infinite while loop, the best way to write it is:

while : ; do <whatever>; done

This has the same effect as using, say, true, but usually has less overhead.

(Commenting things in the Bourne shell has a complicated history that does not fit in this entry.)

What you can't do before you drop setuid permissions

Let us suppose that you have a program that is setuid root but that usually drops its setuid status and reverts to running as the user that ran it. Of course you want to do this as early as possible to reduce the potential security risks, but at the same time you might want do some operations while still root (for reasons of either necessity or convenience in your code structure).

Consider the following tempting sequence, presented in pseudo-code:

chdir(pw->pw_dir);
drop-setuid();

Innocently, one would think that this code is harmless and safe. One would be mistaken; this code is wrong and buggy. If the user's home directory is on NFS and has restricted access permissions, the chdir() will fail, because root has less permissions than a normal user when working over NFS.

Now, consider this sequence (and assume that $HOME is secure, since this is just pseudo-code being written for clarity):

open($HOME/.config, "r");
drop-setuid();

This code is not just buggy, it is a security exposure. It is buggy for the same reason that the first code is (root may not be able to open $HOME/.config, although the user could), and it is a security exposure because the user can make your program open anything on the system with root permissions. Even assuming that there is no way at all to make your program print out any part of the contents of the user's (apparent) configuration file (either correctly formatted or full of syntax errors), there are things that cause side effects when opened and closed.

(And you cannot fix this.)

I don't know what you can safely do before dropping setuid permissions; it's a dangerous and hard problem that I don't know enough about to have an opinion beyond 'as little as possible'. But I do know that you definitely can't do either of these things.

(Now you can probably guess why I was tracing a setuid Linux program, and why I redacted the program name.)