As I noted in Why fork() is a good API, back in the bad old days Unix had a problem of stray file descriptors leaking from processes into commands that they ran (for example, rsh used to gift your shell process with any number of strays). In theory the obvious way to solve this is to have code explicitly close all file descriptors before it exec()s something. In practice Unix has chosen to solve this with a special flag on file descriptors, FD_CLOEXEC, which causes them to be automatically closed when the process exec()s.

In that entry I mentioned that there was a good reason for this alternate solution in practice. At the start of planning this followup entry I had a nice story all put together in my head about why this was so, involving thread-based concurrency races. Unfortunately that story is wrong (although a closely related concurrency race story is the reason for things like O_CLOEXEC in Linux's open()).

FD_CLOEXEC is not necessary to deal with a concurrency race between thread A creating a new file descriptor and thread B fork()ing and then exec()ing in the child process, because the child's file descriptors are frozen at the moment that it's created by fork() (with a standard fork()). It's perfectly safe for the child process to manually close all stray open file descriptors in user-level code, because no matter what it does thread A can never make new file descriptors appear in the child process partway through this. Either they're there at the start (and will get closed by the user-level code), or they'll never be there at all.

There are, however, several practical reasons that FD_CLOEXEC exists. First and foremost, it proved pragmatically easier to get code (often library code) to set FD_CLOEXEC than to get every bit of code that did a fork() and exec() sequence to always clean up file descriptors properly. It also means that you don't have to worry about file descriptors being created in the child process in various ways, especially by library code (which might be threaded code, for extra fun). Finally, it deals with the problem that Unix has no API for finding out what file descriptors your process has open, so your only way of closing all stray file descriptors in user code is the brute force approach of looping trying to close each one in turn (and on modern Unixes, that can be a lot of potential file descriptors).

Once you have FD_CLOEXEC and programs that assume they can use it to just fork() and exec(), you have the thread races that lead you to needing things like O_CLOEXEC. Any time a file descriptor can come into existence without FD_CLOEXEC being set on it, you have a race between thread A creating the file descriptor and then setting FD_CLOEXEC and thread B doing a fork() and exec(). If thread B 'wins' this race, it will inherit a new file descriptor that does not have FD_CLOEXEC set and this file descriptor will leak through the exec().

(All of this is well known in the Unix programming community that pays attention to this stuff. I'm writing it down here so that I can get it straight and firmly fixed into my head, since I almost made an embarrassing mistake about it.)