2010-03-31
Looking back at a year of our disk-based backup system
It's now a bit over a year since we first deployed our disk-based backup system (although I only wrote it up in May, after we built the second machine). This makes it a good moment to look back and talk about how things are going, especially since someone I know asked me just this question recently.
On the whole the answer is that things are going well and quietly; with one exception, there haven't really been any surprises or gotchas. The one exception is that we are seeing somewhat more read errors on the hard drives than we expected or entirely like. It's unlikely to be a bad batch of drives, since we also put some of those drives into our iSCSI backends and they haven't been having anywhere near the same rate of errors.
(We're reasonably careful about handling the disks, including letting them spin down before we remove them from their enclosures, but we do handle them and move them around more than desktop drives probably normally experience. Possibly consumer desktop SATA drives are more sensitive and fragile than one might expect.)
Since we are using these disks strictly for relatively short-term backups, I don't consider this a problem. Things happen to backups all the time; that's why you have more than one backup of anything. Our periodic longer-term archival storage runs are still done to LTO tape (as mentioned in the original entry).
One nice benefit that we didn't entirely expect is that our disk-based backups have drastically speeded up small restores of recently deleted files, which is the most common sort of restore request we get. We can now usually do these in a few minutes (and without having to get up from our desks to go move tapes around), which we quite appreciate.
Remote applications and Gnome settings: an irritation
Here is a conundrum that I have recently run into, posed as a question: when you run a Gnome application remotely through ssh-forwarded X, where does it get its Gnome configuration settings from?
In the old days of X resources, this sort of question had a simple answer; custom settings were attached to the screen itself (they got stuffed into an X property) and thus were accessible to all X programs, regardless of where they were running. In the new ultra-modern world of GConf and friends, the answer is far from obvious.
In theory, Gnome applications get their settings from a per-user GConf daemon, which they talk to via DBus. As usually implemented, DBus is strictly local to the local machine, so the answer should be that they get their configuration settings from whatever Gnome settings you have established on whatever machine the application is running on. (I will not think too hard about how the necessary user GConf daemon gets started; I find it better not to think much about Gnome magic.)
In practice, this seems to work for some settings but not others. In my
testing, Gnome's application specific settings work on a remote machine,
but the general desktop settings do not (even when they influence
application behavior). Specifically, I know that a custom setting for
the cursor_blink
key in /desktop/gnome/interface is ignored when my
session isn't actually on the machine that I'm running gnome-terminal
on. Sadly, this is is kind of a problem for
me.
(In theory this behavior could be justified on the grounds that my
session isn't on that machine, but it results in undesirable behavior
that I have no way to control. And yes, gconftool
on the remote
machine claims that the key is properly set and everything works if my
session actually is native to the remote machine (apart from the bit
where I am using a slow LTSP instead of my nice
desktop machine).)