Wandering Thoughts archives

How I'm trying to do durable disk writes here on Wandering Thoughts

A while back, Wandering Thoughts lost a just posted comment when the host went down abruptly and unexpectedly. Fortunately it was my comment and I was able to recover it, but the whole thing made me realize that I should be doing better about durable writes in DWiki. Up until that point, my code for writing comments to disk had basically been ignoring the issue; I opened a new file, I wrote things, I closed the file, I assumed it was all good. Well, no, you can't really make that assumption.

(Many normal blogs and wikis store things in a database, which worries about doing durable disk writes for you. I am crazy or perverse, so DWiki only uses the filesystem. Each comment is a little file in a particular directory.)

You can do quite a lot of worrying about disk durability on Unix if you try hard (database people have horror stories about how things go wrong). I decided not to worry that much about it, but I did want to do a reasonably competent job. The rules for this on Unix are very complicated and somewhat system dependent, so I am not going to try to cover them; I'm just going to cover my situation. I am writing new files infrequently and I'm willing for this to be a bit slower than it otherwise would be.

Omitting error checking and assuming that you have a fp file and a dirname variable already, what you want in my situation looks like this:

fp.flush()
os.fsync(fp.fileno())

dfd = os.open(dirname, os.O_RDONLY)
os.fsync(dfd)
os.close(dfd)

The first chunk of code flushes the data I've written to the file out to disk. However, for newly created files this is not enough; a filesystem is allowed to not immediately write to disk the fact that the new file is in the directory, and some will do just that. So to be sure we must poke the system so it 'flushes' the directory to disk, ie makes sure that our new file is properly recorded as being present.

(My actual code is already using raw os module writes to write the file, so I can just do 'os.fsync(fd)'. But you probably have a file object, since that's the common case.)

Some sample code you can find online will use os.O_DIRECTORY when opening the directory here. This isn't necessary but doesn't do any harm on most systems as far as I can tell; however, note that there are some uncommon Unix systems that don't even have an os.O_DIRECTORY.

(Although I haven't read through the whole thing and thus can't vouch for it, you may find Reliable file updates with Python to be useful for a much larger discussion of various file update patterns and how to make them reliable. The overall summary here is that reliable file durability on Unix is a minefield. I'm not even sure I have it right, and I try to stay on top of this stuff.)

Sidebar: Why you don't want to use os.sync()

If you're using Python 3.3+, you may be tempted to reach for the giant hammer of os.sync() here; after all, the sync() system call flushes everything to disk and you can't get more sure than that. There are two problems with doing this. First, a full sync() may take a significant length of time, up in the tens of seconds or longer. All you need is for something else to have written but not flushed a bunch of data and the disks to be busy (perhaps your system has backups running) and then boom, your sync() is delaying your program for a very long time.

Second, a full sync() may cause you to trip over unrelated disk problems. If the server you're running on has a completely different filesystem that may be having some disk IO problems, well, your sync() is going to be waiting for the overall system to flush out data to that filesystem even though you don't care about it in the least. I'm unusually sensitive to this issue because I work in an environment with a lot of NFS mounted filesystems that come from an number of different servers, and that means a lot of ways that just one or a few filesystems can be more than a little slow at the moment.

How options in my programs conflict, and where argparse falls short

In my recent entry on argparse I mentioned that it didn't have really top notch handling of conflicting options; instead it only has relatively basic support for this. You might reasonably wonder what it's missing, and thus what top notch argument conflict handling is.

My programs tend to wind up with three sorts of options (command line switches):

• general switches that affect almost everything
• mode-selection switches that pick a major mode of operation
• mode-modifying switches that change how one or more particular major modes work

General switches sometimes conflict with each other (eg --quiet versus --verbose), but apart from that they're applicable all or almost all the time. This is easily represented in argparse with a mutually exclusive group.

Mode selection switches conflict with each other because it normally only makes sense to pick one mode of operation. Some people would make them sub-commands instead (so instead of 'program -L ...' and 'program -Z ...' you'd have 'program op1 ...' and 'program op2 ...'), but I'm a Unix traditionalist and I mostly don't like that approach. Also, often my programs have a default mode where it is just 'program ...'. You can relatively easily represent this in argparse, again with a mutually exclusive group.

(You can't easily handle the case where a general switch happens to be inapplicable to a particular mode, though.)

Mode modifying switches go with one particular mode (or sometimes a couple) and don't make sense without that mode being picked. These logically group with their mode selection switch so that it should be an error to specify them without it. You can't represent this in argparse today; instead you have to check manually, or more likely just allow but silently ignore those switches (because the code paths the program will use doesn't even look at their values).

(And of course you can have nested situations, where some mode modifying switches conflict with each other or spawn sub-modes or whatever.)

It's not hard to see why argparse punts on this. The general case is clearly pretty complicated; you basically need to be able to form multiple arbitrary groups of conflicting arguments and 'these options require this one' sets, and options can be present in multiple groups. Then argparse would have to evaluate all of these constraints to detect conflicts and ideally produce sensible messages about them, which is probably much harder than it looks if there are multiple conflicts.

If I really care about this, I should probably get used to the now fairly common sub-commands approach. Since you can only specify one sub-command you naturally avoid conflicts between them, and argparse can set things up so each sub-command has its own set of options and so on.