Programming Bourne shell scripts is tricky, with dim corners

July 7, 2017

There have been a bunch of good comments on my entry about my views on Shellcheck, so I want to say just a bit too much to fit in a comment of my own. I'll mostly be talking about this script, where I thought the unquoted '$1' was harmless:

echo Error: $1

Leah Neukirchen immediately pointed out something I had completely forgotten, which is that unquoted Bourne shell variable expansion also does globbing. The following two lines give you the same output:

echo *
G="*"; echo $G

This is surprising (to me), but after looking at things I'll admit that it's also useful. It gives you a straightforward way to dynamically construct glob patterns in your shell script and then expand them, without having to resort to hacks that may result in too much expansion or interpretation of special characters.

Then Vidar, the author of Shellcheck, left a comment with an interesting PS suggesting some things that my unquoted use of $1 was leaving me open to:

./testit '-e \x1b[33mTest' # Ascii input results in binary output (when sh is bash)
./testit '-n foo'          # -n disappears

This is a nice illustration of how tricky shell programming can be, because these probably don't happen but I can't say that they definitely don't happen in all Unix environments (and maybe no one can). As bugs, both of these rely on the shell splitting $1 into multiple arguments to the actual command and then echo interpreting the first word (now split into a separate argument) as a -n or -e option, changing its behavior. However, I deliberately wrote testit's use of echo so that this shouldn't happen, as $1 is only used after a non-argument option (the Error: portion).

With almost all commands in a traditional Unix, the first regular argument turns off all further option processing; everything after it will be considered an argument, no matter if it could be a valid option. Using an explicit '--' separator is only necessary if you want your first regular argument to be something that would otherwise be interpreted as an option. However, at least some modern commands on some Unixes have started accepting options anywhere on the command line, not just up to the first regular argument. If echo behaves this way, Vidar's examples do malfunction, with the -n and -e seen as actual options by echo. Having echo behave this way in your shell is probably not POSIX compatible, but am I totally sure that no Unix will ever do this? Of course not; Unixes have done some crazy shell-related things before.

Finally, Aristotle Pagaltzis mentioned, about his reflexive quoting of Bourne shell variables when he uses them:

I’m just too aware that uninvited control and meta characters happen and that word splitting is very complex semantically. [...]

This is very true, as I hope this entry helps illustrate. But for me at least there are three situations in my shell scripts. If I'm processing unrestricted input in a non-friendly environment, yes, absolutely, I had better put all variable usage in quotes for safety, because sooner or later something is going to go wrong. Generally I do and if I haven't, I'd actually like something to tell me about it (and Shellcheck would be producing a useful message here for such scripts).

(At the same time, truly safe defensive programming in the Bourne shell is surprisingly hard. Whitespace and glob characters are the easy case; newlines often cause much more heartburn, partly because of how other commands may react to them.)

If I'm writing a script for a friendly environment (for example, I'm the only person who'll probably run it) and it doesn't have to handle arbitrary input, well, I'm lazy. If the only proper way to run my script is with well-formed arguments that don't have whitespace in them, the only question is how the script is going to fail; is it going to give an explicit complaint, or is it just going to produce weird messages or errors? For instance, perhaps the only proper arguments to a script are the names of filesystems or login names, neither of which have whitespace or funny characters in them in our environment.

Finally, sometimes the code in my semi-casual script is running in a context where I know for sure that something doesn't have whitespace or other problem characters. The usual way for this to happen is for the value to come from a source that cannot (in our environment) contain such values. For a hypothetical example, consider shell code like this:

login=$(awk -F: '$3 == NNN {print $1}' /etc/passwd | sed 1q)
echo whatever $login whatever

This is never going to have problematic characters in $login (for a suitable value of 'never', since in theory our /etc/passwd could be terribly corrupted or there could be a RAM glitch, and yes, if I was going to (say) rm files as root based on this, $login would be quoted just in case).

This last issue points out one of the hard challenges of a Bourne shell linter that wants to only complain about probable or possible errors. To do a good job, you want to recognize as many of these 'cannot be an error' situations as possible, and that requires some fairly sophisticated understanding not just of shell scripting but of what output other commands can produce and how data flows through the script.

By the way, Shellcheck impressed me by doing some of this sort of analysis. For example, it doesn't complain about the following script:

if [ ! -d $ADIR ]; then
   echo does not exist: $ADIR

(If you uncomment the line that sets ADIR from $1, Shellcheck does report problems.)

Comments on this page:

My general attitude is to limit myself to answering “do I have to omit the quotes here?” and going with quotes if not, rather than trying to answer “do I need quotes here?” and leaving them off it not.

The former can be rephrased to “do I need word splitting here?” which is fairly easy to determine, whereas the latter takes far and away more (and more situational) reasoning – yet the operational semantics of the resulting code are identical. And that is if you made no mistakes. Also, all the extra reasoning must be re-performed during every act of maintenance.

I just prefer to conserve my concentration for higher-value decision points. “Operations of thought are like cavalry charges in a battle” and all that.

Of course all that doesn’t change your point about Shellcheck. It can’t even know enough to answer even the more limited question.

By Dan.Astoorian at 2017-07-07 10:41:57:
   ./testit '-e \x1b[33mTest' # Ascii input results in binary output (when sh is bash)
   ./testit '-n foo'          # -n disappears

I'm not sure word splitting actually applies in these cases (variables are expanded after word splitting is performed, as per the bash manpage); I got "Error: -n foo" and "Error: -e \x1b[33mTest" when I actually tried the examples.

Globbing, however, is still very much in play, and that's reason enough to quote the parameter(s).


By Greg A. Woods at 2017-07-10 17:13:30:

I too have learned to do the same thing as Aristotle -- I omit the quotes only if necessary (though old habits of trying to be minimalist in my typing are hard to kick).

BTW, appearance of a pipe in a command line using awk, particularly one designed to produce only one line of output, and especially if the other command(s) is(are) one or more of grep, sed, cut, etc. then it's usually at best a waste of a process and the associated I/O copying, and possibly even a sign of a bug.

Written on 07 July 2017.
« Link: Survey of [floating point] Rounding Implementations in Go
I wish you could easily update the packages on Ubuntu ISO images »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Jul 7 00:00:22 2017
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.