Wandering Thoughts: Recent Entries For 2007/06/12

'Argument list too long' is a misleading message

If you try to run a command with too many command line arguments, you get an 'Argument list too long' error (technically you get an E2BIG error from the kernel, which is then mapped into this error string by your shell). There's a bunch of workarounds for this problem, such as using xargs.

However, the error message is actually somewhat misleading. As old Unix hands know (and new ones generally don't), the kernel's size limit on exec() is on the combination of arguments and environment variables. If you accidentally wind up with a huge environment and try to start a program with even a few arguments, you'll fall over. (And if the environment is big enough, you can't start any programs at all.)

Modern shells make this a more mysterious failure since they have so many built in commands, which means that you can get a substantial way into a shell script before you try to run an external command and fail. The net result is that you can spend a bunch of time scratching your head and trying to figure out why an innocent command with only a couple of arguments is getting this error.

(If you need to figure out what the big environment variables are in the Bourne shell, it is useful to know that the export command without any arguments lists all the environment variables, and it is a built-in command so you can still use it in this situation. If you are trying to use tcsh, you're on your own.)

Sidebar: why I'm so aware of this issue

Most people run into this issue only very rarely, because you need both a very large shell variable or three and to have them exported to the environment. Just pulling an unexpectedly large amount of data into a variable to process it won't hit this. However, I use a shell that automatically exports all shell variables into the environment, so any time I pull a big value into a shell variable I run into this.

My workaround is simple; when I run external commands I null the variable for the command, like so:

BIGVAR='' command ...

This is normally used to add some environment variables just for a command, but it works fine to take them away too.

(This also works in the Bourne shell, although you are much less likely to need it.)