An Amanda backup session involves a lot of processes running on the Amanda client. If you're having a problem with slow backups it can be somewhere between useful and important to understand how everything is connected to everything else and where things go.

A disclaimer: my current information is probably incomplete and it only covers one case. Hopefully it will at least give people an idea of where to start looking and how data is likely to flow around their Amanda setup.

Let's start with an ASCII art process tree of a tar-based backup with indexing (taken from an OmniOS machine with Amanda 3.3.5):

amandad
   sendbackup (1)
      sendbackup (2)
         sh -c '....'
            tar -tf -
            sed -e s/^\.//
      tar --create --file - /some/file/system ....

(In real life the two sendbackup processes are not numbered in pstree or whatever; I'm doing it here to be clear which one I'm talking about.)

Call the 'sh -c' and everything under it the 'indexing processes' and the 'tar --create' the 'backup process'. The backup process is actually making the backup; the indexing processes are reading a copy of the backup stream in order to generate the file index that amrecover will use.

Working outwards from the backup process:

• the backup process is reading from the disk and writing to its standard output. Its standard output goes to sendbackup (2).

• sendbackup (2) reads its standard input, which is from the backup process, and basically acts like tee; it feeds one copy of the data to amandad and another into the indexing processes.

• the indexing processes read standard input from sendbackup (2) and wind up writing their overall standard output to amandad (the tar and sed are in a shell pipeline).

• sendbackup (1) sits there reading the standard error of all processes under it, which I believes it forwards to amandad if it sees any output. This process will normally be not doing anything.

• amandad reads all of these incoming streams of data (the actual backup data, the index data, and the standard error output) and forwards them over the network to your Amanda server. If you use a system call tracer on this amandad, you'll also see it reading from and writing to a pipe pair. I believe this pipe pair is being used for signaling purposes, similar to waiting for both file activity and other notifications.

  (Specifically this is being done by GLib's g_wakeup_signal() and g_wakeup_acknowledge() functions, based on tracing library call activity. I believe they're called indirectly by other GLib functions that Amanda uses.)

Under normal circumstances, everything except sendbackup (1) will be churning away making various system calls, primarily to read and write from various file descriptors. The overall backup performance will be bounded by the slowest component in the whole pipeline of data shuffling (although the indexing processes shouldn't normally be any sort of bottleneck).

Depending on the exact Amanda auth setting you're using, streams from several simultaneous backups may flow from your amandad process to the server either as multiple TCP streams or as one multiplexed TCP stream. See amanda-auth(7) for the full details. In all cases I believe that all of this communication runs through a single amandad process, making it a potential bottleneck.

(At this point I don't know whether amandad is an actual bottleneck for us or something else weird is going on.)

(I knew some of the Amanda internal flow at the time that I wrote ReasoningBackwards, but I don't think I wrote it down anywhere apart from implicitly in that entry and it was in less detail than I do now.)