Thinking about when rsync's incremental mode doesn't help
I mentioned recently that I had
seen cases where
rsync's incremental mode didn't speed it up to any
significant degree. Of course there's an obvious way to create such a
situation, namely erasing and replacing all of the files involved, but
that wasn't it for us. Our case was more subtle and it's taken me a
while to understand why it happened. Ultimately it comes down to having
a subtly wrong mental model of what takes time in
Our specific situation was replicating a mail spool from one machine to
another. There were any number of medium and large inboxes on the mail
spool, but for the most part they were just getting new messages; as
far as we know no one did some major inbox reorganization that would
have changed all of their inbox. Naively you'd think that an
incremental transfer here could go significantly faster than a full
copy; after all, most of what you need to transfer is just the new
messages added to the end of most mailboxes.
What I'm quietly overlooking here is the cost of finding out what
needs to be transferred, and in turn the reason for this is that I've
implicitly assumed that sending things over the network is (very)
expensive in comparison to reading them off the disk. This is an easy
bias to pick up when you work with
purpose is optimizing network transmission and when you use it you
normally don't really think about how it's finding out the differences.
What's going on in our situation is that when
rsync sees a changed
file it has to read the entire file and compute block checksums (on
both sides). It doesn't matter if you've just appended one new email
message to a 100 Mbyte file for a measly 5 Kbyte addition at the end;
rsync still has to read it all. If you have a bunch of midsized
to large files (especially if they're fragmented, as mail inboxes
often are), simply reading through all of the changed files can take a
significant amount of time.
In a way this is a variant of Amdahl's law. With a lot of slightly
changed files an rsync incremental transfer may speed up the network
IO and reduce it to nearly nothing but it can't do much about the
disk IO. Reading lots of data is reading lots of data, whether or
not you send it over the network; you only get a big win out of not
sending it over the network if the network is slow compared to the
disk IO. The closer disk and network IO speeds are to each other,
the less you can save here (and the more that disk IO speeds will
determine the minimum time that an
rsync can possibly take).
The corollary is that where you really save is by doing less disk IO as
well as less network IO. This is where and why things like ZFS snapshots
and incremental '
zfs send' can win big, because to a large extent they
have very efficient ways of knowing the differences that need to be
PS: I'm also making another assumption, namely that CPU usage is free and is not a limiting factor. This is probably true for rsync checksum calculations on modern server hardware, but you never know (and our case was actually on really old SPARC hardware so it might actually have been a limiting factor).