2011-06-19
The Unix shell initialization problem and how shells should work
In general, there are at least three sorts of things that you may want to do when a shell is started: one time environment initializations that can be inherited in subshells, per shell things that cannot be inherited, and anything that you want to do when logging in (you might split this into interactive and non-interactive things under some situations). Depending on what a shell is being started for, you want it to do all, some, or none of these things.
A regular login shell does all of them and then prompts you for
commands; an interactive subshell started as part of your login session
should just do the per shell setup and then prompt for commands. A shell
started when you do 'ssh host command' should do the environment
initialization and then run the command you gave it. An xdm login
shell should do the environment initialization, definitely
skip the interactive login bits, and then either run a command or
execute a shell script (depending on how xdm is set up). And a shell
started via a shell script with '#!/path/to/shell' should normally do
none of these.
(If your shell runs per user dotfiles when executing scripts, it is really hard to create reliable scripts; any user can innocently screw up your script's execution environment in many ways and then your script breaks for them and only them, often oddly. One consequence of this is that if your shell can import things like functions from the environment, you need a switch to turn this behavior off (or just default the import to off when running commands from a file).)
The way Unix shells should work is there should be an agreed on set of command line switches (respected by all shells) to turn on each of the three sorts of things. Programs that needed shells to do certain sorts of setup work could then supply the appropriate switches (and there would be default behaviors, such as doing per shell initialization when started with stdin as a tty). Everyone would be able to be explicit about what was going on and what was being done when; as a bonus we could easily solve the xdm problem, among other issues.
In the real world of Unix as it is today, shells resort to heuristics to tell these situations apart, which occasionally fail. And then we have xdm and its hacks. The situation is extremely unlikely to change now, although in theory a Linux distribution (or several of them) could sit down and bang some heads together over this.
By the way, I maintain that a direct corollary of this whole issue is that your shell should give startup files direct information about what environment the shell is running in (or thinks it's running in). This is especially the case if the shell always or nearly always runs some file. If you do not make this information available you force the authors of startup files to examine the environment and try to reverse engineer your heuristics, which is a losing proposition for any number of reasons.
(The minimum information you provide should be what startup files the shell is planning to run, but more information is better here.)