== Check your backups

Backups have been in the geek news recently, courtesy of the
chilling tale of the online comic Penny Arcade's
[[backup failure|http://www.penny-arcade.com/view.php?date=2005-08-10&res=l]].
Making backups is important, but there's an equally important and far
less appreciated piece: *checking your backups to make sure that you
can actually restore them*.

You may say 'but, I'm sure my system would scream a lot if something
was wrong'. Let me tell you a story about that.

Once upon a time there was a young and innocent system
administrator. He had an Ultrix MIPS DECStation to take care of (which
says something about how long ago this was), and part of taking care
of it was backing it up. Dutifully he arranged tape backups; because
he worked at a university, they were tape backups over the network to
a remote tape drive.

Unfortunately the Ultrix backup program insisted on logging in to the
tape server as the wrong user (and which user it used was
hard-coded). No problem; this was a university, so he had full Ultrix
source code. Changing which user _rdump_ used was a simple text edit
and recompile.

While doing this, he noticed that _rdump_'s _Makefile_ compiled things
without optimization. Since this was on a MIPS-based system, where
compiler optimization was important for decent performance, the system
administrator fixed that when he recompiled _rdump_.

About a year later he accidentally did something to a relatively
unimportant file and decided he wanted to restore it from a backup. He
queued up the right tape, got the _rrestore_ program talking to it,
and _rrestore_ promptly told him that the backup was corrupted. This
could happen sometimes, so he tried another tape; then another; then
all of them. Not one was good.

I will cut to the punchline: the MIPS compiler had an optimizer
bug. Compiling _rdump_ with optimization on ran into this bug (which
was why the original _rdump_ Makefile didn't do that), and the bug
made _rdump_ produced corrupted and unrecoverable output while (of
course) thinking all was fine.

A year's worth of backups were literally worthless. The young system
administrator had a small heart attack, thanked his lucky stars he had
found this before he needed to restore anything important, recompiled
_rdump_ without optimization, and immediately scheduled some full
backups. And tested them afterwards, just to be sure.

So, having [[stubbed my toe|SysadminAphorism]], I strongly urge you:
~~test your backups by trying to restore at least something from
them~~. If you don't, you don't actually know if you have backups, you
just think you do. (And remember, 'optimisim is not a plan'.)