2012-04-20
Sometimes the simple answers are the right ones (a lesson from bash)
A co-worker recently had a cronjob report the following error message, which he asked us for help with:
bash: /a/local/script: /bin/sh: bad interpreter: No such file or directory
At the time that this happened, other messages got logged suggesting that the machine was also apparently under memory pressure.
When I saw this, my mind immediately jumped to ELF interpreters, better known as the dynamic loader. I promptly suggested that maybe the kernel wasn't able to load the dynamic loader for sh, perhaps because of memory pressure or something. However I was wrong, as some web searchers for the error message showed me when I bothered to do them. What's going on is much simpler (although maybe odder) than something complicated about some part of the dynamic loader not working right.
In fact, what's going on is right there in the error message if I had bothered to read it. Here, let me show you with a little test:
$ bash
$ cat /tmp/exmpl
#!/bin/shx
echo hi there
$ /tmp/exmpl
bash: /tmp/exmpl: /bin/shx: bad interpreter: No such file or directory
As odd as it sounds, this error message was almost certainly generated
because (temporarily) there was no /bin/sh. Since /bin/sh is a
magically maintained symlink on many current Linuxes, this is slightly
less odd and peculiar than it seems (it's possible that some package
manipulation made the symlink disappear temporarily), but it's still
pretty odd.
To sum up: bash really was telling my co-worker what was wrong and the error was report was not some peculiar coded message in a bottle that needed a complex, obscure interpretation. The simple answer was the right one. Sometimes that's just how it goes; not everything in system administration is a complex puzzle.
(As it happens I feel that if I'd paid attention to the error message and how it was structured, I would have seen that my complex theory was pretty sure to be wrong. But that's sufficiently tangled to need another entry.)