'Argument list too long' is a misleading message

June 12, 2007

If you try to run a command with too many command line arguments, you get an 'Argument list too long' error (technically you get an E2BIG error from the kernel, which is then mapped into this error string by your shell). There's a bunch of workarounds for this problem, such as using xargs.

However, the error message is actually somewhat misleading. As old Unix hands know (and new ones generally don't), the kernel's size limit on exec() is on the combination of arguments and environment variables. If you accidentally wind up with a huge environment and try to start a program with even a few arguments, you'll fall over. (And if the environment is big enough, you can't start any programs at all.)

Modern shells make this a more mysterious failure since they have so many built in commands, which means that you can get a substantial way into a shell script before you try to run an external command and fail. The net result is that you can spend a bunch of time scratching your head and trying to figure out why an innocent command with only a couple of arguments is getting this error.

(If you need to figure out what the big environment variables are in the Bourne shell, it is useful to know that the export command without any arguments lists all the environment variables, and it is a built-in command so you can still use it in this situation. If you are trying to use tcsh, you're on your own.)

Sidebar: why I'm so aware of this issue

Most people run into this issue only very rarely, because you need both a very large shell variable or three and to have them exported to the environment. Just pulling an unexpectedly large amount of data into a variable to process it won't hit this. However, I use a shell that automatically exports all shell variables into the environment, so any time I pull a big value into a shell variable I run into this.

My workaround is simple; when I run external commands I null the variable for the command, like so:

BIGVAR='' command ...

This is normally used to add some environment variables just for a command, but it works fine to take them away too.

(This also works in the Bourne shell, although you are much less likely to need it.)


Comments on this page:

By Dan.Astoorian at 2007-06-13 10:18:38:

(If you need to figure out what the big environment variables are in the Bourne shell, it is useful to know that the export command without any arguments lists all the environment variables, and it is a built-in command so you can still use it in this situation. If you are trying to use tcsh, you're on your own.)

For csh or tcsh, setenv with no arguments will do it. For tcsh only, printenv is another builtin that will do the job.

BIGVAR='' command ...

The equivalent csh or tcsh idiom requires a subshell (which is okay, since the subshell is merely fork()d, not exec()d) if you only want the variable unset for the one command:

( unsetenv BIGVAR; command ... )

(( setenv BIGVAR ''; command ... ) would be a more literal translation of the Bourne shell idiom, but removing the variable entirely is a little cleaner.)

--Dan

From 84.163.44.45 at 2013-03-30 07:01:18:

Here is some sample code to illustrate the fact that it is a combination of arguments and environment variables that lead to 'Argument list too long' error messages.

http://web.archive.org/web/20110716022320/http://codesnippets.joyent.com/posts/show/12334

http://www.in-ulm.de/~mascheck/various/argmax/

Written on 12 June 2007.
« Getting source RPMs with yumdownloader
What NFS file-based locking problems can happen »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Tue Jun 12 23:15:04 2007
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.