Our backup software was originally written in a world in which you backed up to comparatively slow tape drives and many of its core workings remain in that world even today. One absolutely necessary trick for speeding up tape backups is to first write backups to a 'holding disk' on the backup host, which can be done in parallel, and only later write them serially to tape. Of course if you don't have a holding disk, you must make backups one at a time so you can write them to tape as you make them. Although we back up to disks these days we still do it with Amanda and Amanda still needs a 'holding disk' to do backups in parallel.

On Friday, I noticed that permissions issues meant that one of our backup servers wasn't able to use its holding disk and thus was forced into the much slower mode of only doing one backup at a time. In fact it had been operating this way for some time. More than that, it had been complaining about this for some time, faithfully mentioning the problem in every daily dump report. The mentions looked like this:

driver: WARNING: ignoring holding disk /dumps/holdingdisk: Permission denied

Actually that's not an accurate representation of how they really looked, because I've pulled the one significant line out of over a hundred lines of chatter and I haven't mentioned that this line was down in a section labeled 'Notes', a section that's always full of random unimportant things.

(Amanda backup reports have multiple sections, with more important ones coming earlier. The Notes section is the fourth of five. You can guess how much attention we pay to it based on that.)

This is yet another perfect example of the effects of noise. Amanda put a significant thing in a bunch of noise and we probably spent months not seeing it, because that's what happens when you shower humans with noise; they tune it all out. People don't just tune out the noise, they also tune out any signal that's in the general vicinity and doesn't stand out from the noise enough (generally this is almost all signal).

This reinforces two strong views that I've held for a long time: don't mix signal in with noise and don't generate noise at all. As we've seen here the first doesn't work at all (unless your objective is to hide the signal) and generating noise inevitably contaminates everything around it to some degree.

(I've undoubtedly banged on this particular drum before in other entries. Today I don't feel like hunting them down to add them as links here.)