Some wise words from Henry Spencer on backups

Henry Spencer recently wrote some very useful words of advice on backups on a local sysadmin mailing list. They struck me as the sort of things that are useful enough to share more widely, so with Henry's permission I'm putting his message here. (I thought about running just part of his email, but the more I read it, the more I wanted people to see all of it, so I'm just going to put up the whole thing.)

So, in Henry Spencer's own words:

...So please don't be put off doing a simple thing that will produce significant benefit in most cases, such as storing backups in the next building, just because there exist some "movie plot" scenarios in which this would not be good enough.

I concur. (And I speak as one of the few people on this list who's been running machines on campus long enough to remember the Sandford Fleming fire.) Remember also two things:

(1) A disaster big enough to wipe out both your building and the next building over is likely to have repercussions severe enough to make the up-to-dateness of your offsite backups somewhat secondary.

(2) A wonderful offsite-backup plan which is so inconvenient that it is followed only fitfully is worse than none at all.

There is something to be said for doing an occasional very-offsite backup. But for the weeklies and monthlies, above all you want a plan which is practical enough and convenient enough that you will FOLLOW IT consistently, month after month after month. Hauling a pile of media to and from a remote location gets tedious quickly.

Bear in mind, too, that by a corollary of Murphy's Law, the time when a backup will be most needed will be when the relevant sysadmin is out of town. You want an offsite-backup location that your assistant (etc.) can get access to when necessary; the top shelf of your hall closet is out. If your offsite backups are stored in the next building by informal arrangement between you and the sysadmin there, make sure that other people in both places know about it. You may want to have a formal authorizing letter ("Joe Blow and his staff from Dept. XYZ are authorized to remove or exchange the tapes on the bottom shelf of storage cabinet 3 at any time") on file in case everybody technical at the far end is away.

The one halfway-plausible accident that just might manage to affect two adjacent buildings is a fire. Not because the fire is likely to spread to the second building, but because water and smoke don't necessarily respect building boundaries. (When Sandford Fleming burned down, the firemen spent six hours pouring water in from all sides... and at least one adjacent building was closed due to flooding; indeed, there was flooding as far away as Queen's Park subway station.) Smoke in particular can get into places you'd never think it would reach -- closed drawers, etc. -- and the soot it leaves can be quite corrosive.

There is one simple step you can take that will make your offsite backups much less vulnerable to such indirect hazards: bag them in airtight zip-lock bags. In fact, this is worth doing for the most recent set of on-site backups too -- a serious fire anywhere in your building can expose your computing facility to water and smoke even if the fire never gets anywhere near it.

The hazards of smoke and soot are something I hadn't previously thought of, and the zip-lock bag trick strikes me as both very clever and nicely simple. (I have a weakness for simple, low-tech solutions to problems.)

(PS: for University of Toronto people who stumble over this entry and want to be on the local sysadmins mailing list, you can get on by sending email to ut-admins-request at the domain utcc.utoronto.ca.)