Safely updating Unix files

May 13, 2006

Tim Bray provides Apple some instructions on how to update files containing important data (like iCal calendars). His directions are fine at a high level, but they're actually not quite detailed enough to implement from.

So here's how you safely update Unix files:

  1. create a new scratch file in the same directory as the target file (ideally mode 0600).
  2. write the data to it; optionally fsync(); optionally verify the data.
  3. copy the current version's permissions to the scratch file.
  4. link(2) the current version of the file to the backup file name.
  5. rename(2) the scratch file to the real filename.
  6. fsync() your ex-scratch file, now real file, again.

(Add extra steps to taste if you want to keep more than one backup file.)

The important bit, and the important difference, is the link(2) and the rename(2). This sequence insures that if the system crashes, no matter when, you will always have some version of the file around; you will never be left with foo.old and foo.new but no foo. (This guarantee is often important when doing updating things like system binaries or /etc/passwd.)

For shell scripts, this is:

cp foo /etc/.newfile && \
  cmp -s foo /etc/.newfile && \
  ln -f /etc/foo /etc/foo.bak && \
  mv -f /etc/.newfile /etc/foo
sync

I see a lot of shell scripts that leave out the cmp. I always think that they're being awfully optimistic about cp's error handling. (Technically you should have a sync between the cp and the cmp because that's the only insurance you have against physical disk errors on the write; what cp has written has not necessarily hit the platters and thus run into any physical media problems by the time it exits successfully. Issues like this give paranoid sysadmins nervous tics.)

I have omitted the step to synchronize permissions and ownership, because I don't know of a good utility to copy them from file A to file B.


Comments on this page:

By DanielMartin at 2006-05-13 09:08:51:

The chown and chmod commands on most linux variants accept a --reference option that will copy the owner/permission from another file.

I'd be very surprised if you can find an implementation of cp such that cp will exit successfully but cmp will fail without there being a sync in between. (In other words, I see the point of the cmp, but don't see any point in using cmp without a sync)

From 70.53.120.146 at 2006-05-13 13:10:47:

Not only these steps are important in the event of a system crash, but also if the files being updated are in use by any program. The "ln + mv" steps guarantee that the old file is not overwritten (unlike "cp").

Many Unix admins have probably learned the "ln + mv" steps the hard way, like when trying to manually update libc.so or other critical system files while the system is running, and happily trying "cp libc.so.new /usr/lib/libc.so". Either the cp fails (/bin/cp needs libc.so) and you end up with an empty libc.so, and/or every other program using libc.so crashes, which is pretty much the whole system.

Not too long ago I fell for this while making minor changes to /etc/procmailrc and noticing weird errors in e-mails that were arriving at the exact moment I saved the file (fortunately without loss). Then I wrote a script like what you describe to call "vi" on a procmailrc.temp file then safely replace the original.

Oscar@MIE

Written on 13 May 2006.
« The problem with preforking Python network servers
Weekly spam summary on May 13th, 2006 »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sat May 13 01:06:33 2006
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.