The roots of an obscure Bourne shell error message

September 10, 2023

Suppose that you're writing Bourne shell code that involves using some commands in a subshell to capture some information into a shell variable, 'AVAR=$(....)', but you accidentally write it with a space after the '='. Then you will get something like this:

$ AVAR= $(... | wc -l)
sh: 107: command not found

So, why is this an error at all, and why do we get this weird and obscure error message? In the traditional Unix and Bourne shell way, this arises from a series of decisions that were each sensible in isolation.

To start with, we can set shell variables and their grown up friends environment variables with 'AVAR=value' (note the lack of spaces). You can erase the value of a shell variable (but not unset it) by leaving the value out, 'AVAR='. Let's illustrate:

$ export FRED=value
$ printenv | fgrep FRED
$ printenv | fgrep FRED
$ unset FRED
$ printenv | fgrep FRED
$ # ie, no output from printenv

Long ago, the Bourne shell recognized that you might want to only temporarily set the value of an environment variable for a single command. It was decided that this was a common enough thing that there should be a special syntax for it:

$ PATH=/special/bin:$PATH FRED=value acommand

This runs 'acommand' with $PATH changed and $FRED set to a value, without changing (or setting) either of them for anything else. We have now armed one side of our obscure error, because if we write 'AVAR= ....' (with the space), the Bourne shell will assume that we're temporarily erasing the value of $AVAR (or setting it to a blank value) for a single command.

The second part is that the Bourne shell allows commands to be run to be named through indirection, instead of having to be written out directly and literally. In Bourne shell, you can do this:

$ cmd=echo; $cmd hello world
hello world
$ cmd="echo hi there"; $cmd
hi there

The Bourne shell doesn't restrict this indirection to direct expansion of environment variables; any and all expansion operations can be used to generate the command to be run and some or all of its arguments. This includes subshell expansion, which is written either as $(...) in the modern way or as `...` in the old way (those are backticks, which may be hard to see in some fonts). Doing this even for '$(...)' is reasonably sensible, probably sometimes useful, and definitely avoids making $(...) a special case here.

So now we have our perfect storm. If you write 'AVAR= $(....)', the Bourne shell first sees 'AVAR= ' (with the space) and interprets it as you running some command with $AVAR set to a blank value. Then it takes the '$(...)' and uses it to generate the command to run (and its command line). When your subshell prints out its results, for example the number of lines reported by 'wc -l', the Bourne shell will try to use that as a command and fail, resulting in our weird and obscure error message. What you've accidentally written is similar to:

$ cmd=$(... | wc -l)
$ AVAR= $cmd

(Assuming that the $(...) subshell doesn't do anything different based on $AVAR, which it probably doesn't.)

It's hard to see any simple change in the Bourne shell that could avoid this error, because each of the individual parts are sensible in isolation. It's only when they combine together like this that a simple mistake compounds into a weird error message.

(The good news is that shellcheck warns about both parts of this, in SC1007 and SC2091.)

Comments on this page:

I've been caught by this one before. It feels so natural to put spaces around equals signs.

Only today I encountered this fun issue that is totally shell (not python!) related:

 #!/usr/bin/env python3
from dataclasses import dataclass
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt 

Here is a copy to download, mark +x and run (research shows that random executables from the internet are good for your health). If you do it correctly then you'll get this error message:

 ./ line 3: from: command not found

... and your cursor will turn into a cross.


Spoiler: there is a sneaky space character before the #! on the first line that got there when I copied and pasted the header into this script. The kernel doesn't know what to do with this executable (the #! shebang is "missing" because of that space before it), so it runs it as a shell script. "from" isn't a valid command and "import" is the screenshot utility from imagemagick which changes your cursor until you click on a window or drag a box.

I'm glad I can spot and understand these issues quickly nowadays, I have these vague memories of losing entire weeks to other insane single-character stuff like having a semicolon after a while () in C. Everything compiles and runs, just not the way you'd expect :D

By Miksa at 2023-09-11 08:28:31:

There is a similar issue that Nordic/European shell users may encounter. You input any command and try to pipe it to less and you end up with:

ls -lF | less
bash: less: command not found

Yet 'less' works as it always has. You get this randomly and you may be confused about it for a long time until you figure out the issue. On Nordic keyboard layouts you input "|" by AltGr+<, the right Alt. A quick typist may follow this with AltGr+Space, which on some configurations produces a non-breaking-space.

When I built my customized keyboard layout I made sure I could input pipe with a single keypress with my thumb.

By tux0r at 2023-09-12 20:54:33:

Wasn't $(...) a bashism, mostly? I thought backticks were the only way supported by the Bourne (and almost every other contemporary) shell.

By cks at 2023-09-13 11:45:27:

I had remembered $(...) as a POSIX thing, but it turns out that the POSIX shell specification is a subset of the Korn shell, so $(...) originated there and spread out afterward. I'm not sure when various Bourne shell variants started picking it up, but I'm sure that POSIX standardizing it helped (once people started making their Bourne shell version be POSIX compliant). I suspect that any Bourne shell implementation that was started post-POSIX would have implemented $(...) early on, and other Bourne shell versions grew it over time.

Written on 10 September 2023.
« The effects of modest TCP latency (I think) on my experience with some X programs
GNU Emacs, use-package, and key binding for mode specific keymaps »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun Sep 10 22:12:44 2023
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.