The cause of our slow Amanda backups and our workaround
A while back I wrote about the challenges in diagnosing slow (Amanda) backups. It's time for a followup entry on that, because we found what I can call 'the problem' and along with it a workaround. To start with, I need to talk about how we had configured our Amanda clients.
In order to back up our fileservers
in a sensible amount of time, we run multiple backups on each them
at once. We don't really try to do anything sophisticated to balance
the load across multiple disks both because this is hard in our
environment (especially given limited Amanda features) and because
we've never seen much evidence that reducing overlaps was useful
in speeding things up; instead we just have Amanda run three backups
at once on each fileserver ('
maxdumps 3' in Amanda configuration).
For historical reasons we were also using Amanda's '
auth bsd' style
of authentication and communication.
As I kind of mentioned in passing in my entry on Amanda data flows, '
auth bsd' communication causes all
concurrent backup activity to flow through a single master
process. It turned out that this was our bottleneck. When we had a
amandad process handling sending all backups back to the
Amanda server and it was running more than one filesystem backup
at a time, things slowed down drastically and we experienced our
problem. When an
amandad process was only handling a single backup,
things went fine.
We tested and demonstrated this in two ways. The first was we dropped
one fileserver down to one dump at a time and then it ran fine. The
more convincing test was to use
SIGCONT to pause and then resume backups
on the fly on a server running multiple backups at once. This
demonstrated that network bandwidth usage jumped drastically when
we paused two out of the three backups and tanked almost immediately
when we allowed more than one to run at once. It was very dramatic.
Further work with a DTrace script
provided convincing evidence that it was the
itself that was the locus of the problem and it wasn't that, eg,
tar reads slowed down drastically if more than one
running at once.
Our workaround was to switch to Amanda's '
auth bsdtcp' style of
communication. Although I initially misunderstood what it does, it
turns out that this causes each concurrent backup to use a separate
amandad process and this made everything work fine for us;
performance is now up to the level where we're saturating the
backup server disks instead of the network.
Well, mostly. It turns out that our first-generation ZFS fileservers probably also have the slow backup
problem. Unfortunately they're running a much older Amanda version
and I'm not sure we'll try to switch them to '
auth bsdtcp' since
they're on the way out anyways.
I call this a workaround instead of a solution because in theory a
amandad process handling all backup streams shouldn't
be a problem. It clearly is in our environment for some reason, so it
sort of would be better to understand why and if it can be fixed.
(As it happens I have a theory for why this is happening, but it's
long enough and technical enough that it needs another entry. The short version is that
I think the
amandad code is doing something wrong with its socket
Does init actually need to do daemon supervision?
Sure, init has historically done some sort of daemon supervision (or at least starting and stopping them) and I listed it as one of init's jobs. But does it actually need to do this? This is really two questions and thus two answers.
Init itself, PID 1, clearly does not have to be the process that does daemon supervision. We have a clear proof of this in Solaris, where SMF moves daemon supervision to a separate set of processes. SMF is not a good init system but its failures are failures of execution, not of its fundamental design; it does work, it's just annoying.
Whether the init system as a whole needs to do daemon supervision is a much more philosophical question and thus harder to answer. However I believe that on the whole the init system is the right place for this. The pragmatics of why are simple: the init system is responsible for booting and shutting down the system and doing this almost always needs at least some daemons to be started or stopped in addition to more scripted steps like filesystem checks. This means that part of daemon supervision is at least quite tightly entwined with booting, what I called infrastructure daemons when I talked about init's jobs. And since your init system must handle infrastructure daemons it might as well handle all daemons.
(In theory you could define an API for communication between the init system and a separate daemon supervision system in order to handle this. In practice, until this API is generally adopted your init system is tightly coupled with whatever starts and stops infrastructure daemons for it, ie you won't be able to swap one infrastructure daemon supervision system for another and whichever one your init system needs might as well be considered part of the init system itself.)
I feel that the pragmatic argument is also the core of a more philosophical one. There is no clear break between infrastructure daemons and service daemons (and in fact what category a daemon falls into can vary from system to system), which makes it artificial to have two separate daemon supervision systems. If you want to split the job of an init system apart at all, the 'right' split is between the minimal job of PID 1 and the twin jobs of booting the system and supervising daemons.
(This whole thing was inspired by an earlier entry being linked to by this slashdot comment, and then a reply to said comment arguing that the role of init is separate from a daemon manager. As you can see, I don't believe that it is on Unix in practice.)
Sidebar: PID 1 and booting the system
This deserves its own entry to follow all of the threads, but the simple version for now: in a Unix system with (only) standard APIs, the only way to guarantee that a process winds up as PID 1 is for the kernel to start it as such. The easiest way to arrange for this is for said process to be the first process started so that PID 1 is the first unused PID. This naturally leads into PID 1 being responsible for booting the system, because if it wasn't the kernel would have to also start another process to do this (and there would have to be a decision about what the process is called and so on).
This story is increasingly false in modern Unix environments which do various amounts of magic setup before starting the final real init, but there you have it.