2009-02-09
Backups and archives
When I am thinking of backup issues, I tend to strongly separate backups from archives, and in fact I believe that there are two different sorts of backups with different characteristics. Since I think that this is an important issue, let me explain my distinctions.
To start with, to me the difference between backups and archives is that backups are for recovering after a system failure, while archives are for storing data that you no longer keep online. Here, you have to consider 'system failure' somewhat broadly, so that it includes not just the disks melting down but also things like users (or automated programs, or sysadmin error) accidentally removing or damaging files.
Within backups, there are two sorts of backups: short term backups and long term backups. A short term backup is one that you can pretty much count on restoring on to exactly the same sort of system, the same system environment, that you have now. A long term backup is a backup where you cannot count on this; you are keeping the backup for long enough that you may have done things like upgraded operating system versions and cannot now build an 'old' system just to restore some data.
If you have short term backups, it's perfectly acceptable to use a
convenient data format that is very system specific (for example, ZFS's
zfs send). But if you have long term backups, you run into potential
issues with such things (for example, zfs send is explicitly not
guaranteed to work across different Solaris versions), and you need a
more portable backup format. How much more portable depends on how much
of a radical change you expect, which also depends on how far back you
need to be able to go. If you need to go far enough back, you get into
archiving territory with all of the headaches that that implies.
(The other thing you need for long term backups is to keep track of how things got relocated and shuffled around in your filesystems, so that you can easily find out where they were in the past in order to do restores. For example, can you easily recover where a user's home directory or a particular database backup was six months ago?)
The archive versus backup distinction is especially important because true archiving is very hard, fundamentally because archiving has very ambitious goals and represents a total commitment (because the data exists only in the archives). Backups are much simpler because they have much smaller goals and generally a much shorter time span, so they do not have to be anywhere as durable (both in media and in the capability to do something useful with the media).
It is my personal belief that archiving is different enough from backups that you should not try to do real archiving with the same system that you use for backups. But if you actually want to use your backups as archives too, you must explicitly design for this and consider the archiving problems. You cannot just assume that your backup format, approach, or system makes for a good archival system, especially as you want to go further and further back in time.
Sidebar: why you want multiple backups
As a side note, somewhat in reaction to something I read recently, I do not feel that backups turn into archives just because you have more than one of them. You should make and keep multiple backups because you want insurance against all of the different sorts of backup failures, which include media failures, backup software failures, and failures to notice damage immediately.
Archives need insurance too, but it takes different forms: you make multiple copies of the archives, and you may do so on different sorts of media or in different formats in case one form turns out to be less durable than expected.
(There are people who make multiple copies of individual backups, but they tend to be either exceptionally paranoid or working with exceptionally high-value data.)
I hate flaky systems, Fedora 10 and/or hardware edition
I am generally an even, calm person. But there are some things that just absolutely get under my skin and drive me to screaming frustration, and one of them is flaky, unreliable systems. This is especially the case when I can't tell if the problem is hardware, software, or both.
Since I'm writing about this, you might guess that I am dealing with such a thing right now. You would be correct; my office workstation has become really flaky ever since I upgraded it from Fedora 8 to Fedora 10 (about a month ago), with frequent kernel panics and other problems. In a sysadmin's workstation, this is more than frustrating, it's actively dangerous; I've already had the system crash once while I was in the middle of something moderately delicate.
According to the Fedora reply to my bug report, this is almost certainly a hardware problem. I wish I could entirely believe and trust that, but it's hard since the system was completely stable on Fedora 8, with not a crash or problem in sight. And without a good idea of where the problem is, it's hard to go to management with a request for, say, new hardware, since we might be spending money and getting nothing for it.
(This is where it is frustrating that universities are not businesses. In many companies, the cost of my time means that it would long since have made economic sense to just get me a new machine and put the old one in the 'possibly junk' spares pool. But this is not something that universities do, especially in the current budget climate; staff are a sunk cost, while hardware costs real money.)
Unfortunately falling back to Fedora 8 is not a real option at this point; I don't have an alternate OS install on this machine, and while Fedora lets you upgrade a machine, there's no downgrade procedure. I will probably try to wedge the current Fedora 8 kernel into the system and see if I can boot it without everything breaking.
(At a look, the only thing that seems to specifically depend on a recent kernel version is the ATI graphics driver. Unfortunately having working X is somewhat important to me, so I am hoping that it will work in 'nomodeset' mode.)
PS: yes, I have a pre-upgrade backup of my machine, since I'm not crazy. But falling back to it at this stage would take a bunch of work for various reasons, including trying to hunt down various things that have changed since then.