How data flows around on the client during an Amanda backup
An Amanda backup session involves a lot of processes running on the Amanda client. If you're having a problem with slow backups it can be somewhere between useful and important to understand how everything is connected to everything else and where things go.
A disclaimer: my current information is probably incomplete and it only covers one case. Hopefully it will at least give people an idea of where to start looking and how data is likely to flow around their Amanda setup.
Let's start with an ASCII art process tree of a tar-based backup with indexing (taken from an OmniOS machine with Amanda 3.3.5):
amandad sendbackup (1) sendbackup (2) sh -c '....' tar -tf - sed -e s/^\.// tar --create --file - /some/file/system ....
(In real life the two sendbackup processes are not numbered
pstree or whatever; I'm doing it here to be clear which
one I'm talking about.)
Call the '
sh -c' and everything under it the 'indexing processes'
and the '
tar --create' the 'backup process'. The backup process
is actually making the backup; the indexing processes are reading
a copy of the backup stream in order to generate the file index
amrecover will use.
Working outwards from the backup process:
- the backup process is reading from the disk and writing to its
standard output. Its standard output goes to sendbackup (2).
- sendbackup (2) reads its standard input, which is from the backup
process, and basically acts like
tee; it feeds one copy of the data to amandad and another into the indexing processes.
- the indexing processes read standard input from sendbackup (2)
and wind up writing their overall standard output to amandad (the
sedare in a shell pipeline).
- sendbackup (1) sits there reading the standard error of all
processes under it, which I believes it forwards to amandad if
it sees any output. This process will normally be not doing
- amandad reads all of these incoming streams of data (the actual
backup data, the index data, and the standard error output) and
forwards them over the network to your Amanda server. If you use
a system call tracer on this amandad, you'll also see it reading
from and writing to a pipe pair. I believe this pipe pair is being
used for signaling purposes, similar to waiting for both file
activity and other notifications.
(Specifically this is being done by GLib's
g_wakeup_acknowledge()functions, based on tracing library call activity. I believe they're called indirectly by other GLib functions that Amanda uses.)
Under normal circumstances, everything except sendbackup (1) will be churning away making various system calls, primarily to read and write from various file descriptors. The overall backup performance will be bounded by the slowest component in the whole pipeline of data shuffling (although the indexing processes shouldn't normally be any sort of bottleneck).
Depending on the exact Amanda
auth setting you're using, streams
from several simultaneous backups may flow from your amandad process
to the server either as multiple TCP streams or as one multiplexed
TCP stream. See
amanda-auth(7) for the full details. In all cases
I believe that all of this communication runs through a single
amandad process, making it a potential bottleneck.
(At this point I don't know whether amandad is an actual bottleneck for us or something else weird is going on.)
(I knew some of the Amanda internal flow at the time that I wrote ReasoningBackwards, but I don't think I wrote it down anywhere apart from implicitly in that entry and it was in less detail than I do now.)