Wandering Thoughts archives

2016-04-06

What is behind Unix's 'Text file is busy' error

Perhaps you have seen this somewhat odd Unix error before:

# cp prog /usr/local/bin/prog
cp: cannot create regular file 'prog': Text file is busy

This is not just an unusual error message, it's also a rare instance of Unix being friendly and not letting you blow your foot off with a perfectly valid operation that just happens to be (highly) unwise. To understand it, let's first work out what exact operation is failing. I'll do this with strace on Linux, mostly because it's what I have handy:

$ cp /usr/bin/sleep /tmp/
$ /tmp/sleep 120 &
$ strace cp /usr/bin/sleep /tmp/
[...]
open("/usr/bin/sleep", O_RDONLY)        = 3
fstat(3, {st_mode=S_IFREG|0755, st_size=32600, ...}) = 0
open("/tmp/sleep", O_WRONLY|O_TRUNC)    = -1 ETXTBSY (Text file busy)
[...]

There we go. cp is failing when it attempts to open /tmp/sleep for writing and truncate it, which we have a running program, and the specific Unix errno value here is ETXTBSY. If you experiment some more you'll discover that we're allowed to remove /tmp/sleep if we want to, just not write to it or truncate it (at least on Linux; the specifics of what's disallowed may vary slightly on other Unixes). This is an odd limitation for Unix, because normally there's nothing that prevents one process from modifying a file out from underneath another process (even in harmful ways). Unix leaves it up to the program(s) involved to coordinate things between themselves, rather than enforcing a policy of 'no writing if there are readers' or something in the kernel.

But running processes are special, because really bad things usually happen if you modify the on-disk code of a running process. The problem is virtual memory, or more exactly paged virtual memory. On a system with paged virtual memory, programs aren't loaded into RAM all at once and then kept there; instead they're paged into RAM in bits and pieces as bits of code (and data) are needed. In fact, some times already-loaded bits and pieces are dropped from RAM in order to free up space, since they can always be loaded back in from disk.

Well, they can be loaded back in from disk if some joker hasn't gone and changed them on disk, at least. All of this paging programs into RAM in sections only works if the program's file on disk doesn't ever change while the program is running. If the kernel allowed running programs to change on disk, it could wind up loading in one page of code from version 1 of the program and another page from version 2. If you're lucky, the result would segfault. If you're unlucky, you might get silent malfunctions, data corruption, or other problems. So for once the Unix kernel does not let you blow your foot off if you really want to; instead it refuses to let you write to a program on disk if the program is running. You can truncate or overwrite any other sort of file even if programs are using it, just not things that are part of running programs. Those are special.

Given the story I've just told, you might expect ETXTBSY to have appeared in Unix in or around 3BSD, which is more or less the first version of Unix with paged virtual memory. However, this is not the case. ETXTBSY turns out to be much older than BSD Unix, going back to at least Research V5. Research Unix through V7 didn't have paged virtual memory (it only swapped entire programs in and out), but apparently the Research people decided to simplify their lives by basically locking the files for executing programs against modification.

(In fact Research Unix was stricter than modern Unixes, as it looks like you couldn't delete a program's file on disk if it was running. That section of the kernel code for unlink() gets specifically commented out no later than 3BSD, cf.)

PS: the 'text' in 'text file' here actually means 'executable code', per say size's output. Of course it's not just the actual executable code that could be dangerous if it changed out from underneath a running program, but there you go.

Sidebar: the way around this if you're updating running programs

To get around this, all you have to do is remove the old file before writing the new file into place. This (normally) doesn't cause any problems; the kernel treats the 'removed but still being used by a running program' executable the same way it treats any 'removed but still open' file. As usual the file is only actually removed when the last reference goes away, in this case the last process using the old executable exits.

(Of course NFS throws a small monkey wrench into things, sometimes in more than one way.)

unix/WhyTextFileBusyError written at 23:05:38; Add Comment

How options in my programs conflict, and where argparse falls short

In my recent entry on argparse I mentioned that it didn't have really top notch handling of conflicting options; instead it only has relatively basic support for this. You might reasonably wonder what it's missing, and thus what top notch argument conflict handling is.

My programs tend to wind up with three sorts of options (command line switches):

  • general switches that affect almost everything
  • mode-selection switches that pick a major mode of operation
  • mode-modifying switches that change how one or more particular major modes work

General switches sometimes conflict with each other (eg --quiet versus --verbose), but apart from that they're applicable all or almost all the time. This is easily represented in argparse with a mutually exclusive group.

Mode selection switches conflict with each other because it normally only makes sense to pick one mode of operation. Some people would make them sub-commands instead (so instead of 'program -L ...' and 'program -Z ...' you'd have 'program op1 ...' and 'program op2 ...'), but I'm a Unix traditionalist and I mostly don't like that approach. Also, often my programs have a default mode where it is just 'program ...'. You can relatively easily represent this in argparse, again with a mutually exclusive group.

(You can't easily handle the case where a general switch happens to be inapplicable to a particular mode, though.)

Mode modifying switches go with one particular mode (or sometimes a couple) and don't make sense without that mode being picked. These logically group with their mode selection switch so that it should be an error to specify them without it. You can't represent this in argparse today; instead you have to check manually, or more likely just allow but silently ignore those switches (because the code paths the program will use doesn't even look at their values).

(And of course you can have nested situations, where some mode modifying switches conflict with each other or spawn sub-modes or whatever.)

It's not hard to see why argparse punts on this. The general case is clearly pretty complicated; you basically need to be able to form multiple arbitrary groups of conflicting arguments and 'these options require this one' sets, and options can be present in multiple groups. Then argparse would have to evaluate all of these constraints to detect conflicts and ideally produce sensible messages about them, which is probably much harder than it looks if there are multiple conflicts.

If I really care about this, I should probably get used to the now fairly common sub-commands approach. Since you can only specify one sub-command you naturally avoid conflicts between them, and argparse can set things up so each sub-command has its own set of options and so on.

python/ArgparseAndHowOptionsConflict written at 01:06:50; Add Comment


Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.