2014-03-19
Why I like ZFS's zfs send and zfs receive
Our new fileserver environment to be has reached the point where it needs real testing, so on Monday I took the big leap and moved my home directory filesystem to our first new fileserver in order to give it some real world usage. Actually that's a bit of a misnomer. I'd copied my home directory filesystem over to the new fileserver several weeks ago simply to put some real data into the system; what I did on Monday was re-synchronize the copy with the live version and then switch all of our actual Ubuntu servers to NFS mounting it from the new fileserver.
This is exhibit one of why I like zfs send and zfs receive, or more
specifically why I like incremental zfs send and zfs receive. They
make it both easy and efficient to copy, repeatedly update, and then
synchronize filesystems like this during a move. You can sort of do the
same thing with rsync at the user level but zfs send does it faster
(sometimes drastically so), to some degree more reliably, and certainly
more conveniently.
(As I've found the hard way, if there is enough of the wrong sort of
activity in a filesystem an incremental rsync can take very close to
the same amount of time as a non-incremental one. That was a painful
experience. This doesn't happen with zfs send; you only ever pay for
the amount of data actually changed.)
The other reason why I'm so fond of zfs send is this magic trick:
incremental zfs send with snapshots is symmetric. Once you've
synchronized two copies this way you can do it in either direction and
you can reverse directions partway through. My home directory's move
is almost certainly temporary (and it's certainly temporary if problems
arise) and as long as I retain the filesystems and snapshots on both
sides I can move back just as fast as I moved in the first place. I
don't have to make a full copy and then synchronize it and so on; I
can just make a new snapshot on the new fileserver and send back the
difference between my last 'move over' snapshot and this first 'move
back' snapshot. Speaking as someone who's currently basically jumping up
and down on a new fileserver that should be good but hasn't been fully
proven yet, that's a rather reassuring thing.
(In fact if I'm cautious I can update my old home directory every night or just periodically, so that I won't lose much even if the new environment goes down in total flames somehow.)
PS: Since I've now tested this in both directions, I can say that
you can zfs send a filesystem from Solaris 10 update 8 to OmniOS
and then zfs send snapshots from it from OmniOS back to S10U8
(provided that you don't 'zfs upgrade' the filesystem while it's
on OmniOS; if you do, it'll become incompatible with S10U8's old
ZFS version (or at least so the documentation says, I haven't tested
this personally for the obvious reasons)).
Sidebar: how we freeze filesystems when moving them
The magic sequence we use is to unshare the filesystem (since all
of our access to filesystems is over NFS), set readonly=on, and
then make the final move snapshot. After the newly moved filesystem
is up and running we set canmount=off on the old version and then
let it sit until we have a backup of the moved one and everything
is known to be good and so on.
We almost always do zfs receive into unmounted filesystems (ie
'zfs receive -u ...'.)