How I shot my foot because the Bourne shell is different

July 5, 2017

I have an invariable, reflexive habit in the Bourne shell, which is that I call my for loop variables $i. This reflex is so ingrained that if I try to fight it, I can wind up writing loops that look like this:

for avar in ....; do
    somecommand $i
done

I may have carefully written the for loop using a sensible, non-$i variable name, but then when I was writing the body of the loop I forgot and reflexively used $i. This is always a fun, forehead-slapping issue to debug (even if it often doesn't take long, especially in tiny for loops).

(Much of this isn't unique to the Bourne shell; like any number of people, I normally use i as my default loop variable name no matter what language I'm working in.)

Recently I wrote a script where the top level looked something like this:

... various functions ...
reporton() { ... }

for i in $@; do
    case "$i" in
       magic) reporton "magic $i" $(magic-thing $i) ;;
       ...) ... ;;
       /cs/htuser/*) reporton "$i" $(webdir-fs $i) ;;
       ...) ... ;;
       *) # assume username
          reporton "<$i>" $(user-fs $i)
          reporton "<$i> other" $(webdir-fs /cs/htuser/$i) ;;
    esac
 done

Most of the various arguments I could give the script worked fine. In the username case, the first reporton worked properly but then the second one failed mysteriously, printing out weird messages. To make it more puzzling, the same reporton worked fine when run independently (in the /cs/htuser/* case).

It took a bit of time before the penny dropped. Several of my shell functions had their own for loops, and of course I had reflexively written them using $i as the loop variable. Since I was writing this in more or less pure Bourne shell, I wasn't using 'local i' in any of these functions and so everyone's for loops were changing the same global $i variable.

For most of the functions, this worked out; they didn't call other $i-changing functions inside their own for loops, so the value of $i was stable in their for loop bodies. But at the top level this wasn't the case; I was obviously calling the whole stack of functions and was having $i's value changed out from underneath me. Most of the time this didn't matter because I only used $i once (before its value got changed by the functions I called). The 'assume username' case was the exception; as you can see, I ran two reporton's in succession and used $i with each. When I got to the second reporton, $i's value was completely mangled and things blew up.

Most languages that I deal with are lexically scoped languages, where reusing the same name for variables in different scopes just works (in that each version of the variable name is completely independent). Lexical scoping is so pervasive in my programming languages that I think of it as the normal, default case. The Bourne shell is one of the few exceptions; it's dynamically scoped, and so all of the $i's are the same variable and my various usages of $i were stepping on each other. Since it's the rare exception and I don't do complicated Bourne shell programming very often, I completely forgot this difference when I wrote my script. Hopefully I'll now remember it for the next time I write something in the Bourne shell that's sufficiently complicated that it uses functions and multiple for loops.

(Awk is another language that I deal with that normally has dynamic scope, but I've only ever written a few pieces of awk that were complicated enough to use functions (such as this one).)


Comments on this page:

By skeeto at 2017-07-05 09:14:38:

You should wrap that $@ in quotes so that it doesn't word split IFS within each argument. Quoting with "$@" is a special case in Bourne shell.

   for i in "$@"; do
       ...
   done
By cks at 2017-07-05 12:51:14:

It turns out that the actual script uses the fully correct version of 'for i in "$@"; do'; I just absently left out the quotes around $@ when I wrote the entry and was (re)writing an abstracted version of the top level loop.

(It doesn't matter for this particular script because no valid argument to it will ever have spaces in it, and it's only for my own internal use. But always using "$@" in this sort of context is a good habit to have.)

By Anon at 2017-07-05 15:30:42:

Have you tried ShellCheck for your bash snippets - http://www.shellcheck.net/ ?

By dwfreed at 2017-07-05 20:04:22:

Declare all the variables you use in a function that shouldn't leave that function as local, eg:

reporton() {
  local i

  for i in foo; do
    stuff
  done
}

You can also use local on exported variables, and your changes will only apply within that function:

foobar() {
  local TERM=dummy
  env | grep TERM=
}

$ export TERM=xterm
$ env | grep TERM=
TERM=xterm
$ foobar
TERM=dummy
$ env | grep TERM=
TERM=xterm
By Twirrim at 2017-07-05 22:48:20:

Have you tried giving shellcheck a run at your scripts? https://github.com/koalaman/shellcheck It's been catching some interesting potential bugs in stuff. You can hook it in with vim via https://github.com/vim-syntastic/syntastic

Written on 05 July 2017.
« LinkedIn is still trying to send me email despite years of rejections
My current views on Shellcheck »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Wed Jul 5 01:25:37 2017
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.