Why I like ZFS's zfs send and zfs receive

March 19, 2014

Our new fileserver environment to be has reached the point where it needs real testing, so on Monday I took the big leap and moved my home directory filesystem to our first new fileserver in order to give it some real world usage. Actually that's a bit of a misnomer. I'd copied my home directory filesystem over to the new fileserver several weeks ago simply to put some real data into the system; what I did on Monday was re-synchronize the copy with the live version and then switch all of our actual Ubuntu servers to NFS mounting it from the new fileserver.

This is exhibit one of why I like zfs send and zfs receive, or more specifically why I like incremental zfs send and zfs receive. They make it both easy and efficient to copy, repeatedly update, and then synchronize filesystems like this during a move. You can sort of do the same thing with rsync at the user level but zfs send does it faster (sometimes drastically so), to some degree more reliably, and certainly more conveniently.

(As I've found the hard way, if there is enough of the wrong sort of activity in a filesystem an incremental rsync can take very close to the same amount of time as a non-incremental one. That was a painful experience. This doesn't happen with zfs send; you only ever pay for the amount of data actually changed.)

The other reason why I'm so fond of zfs send is this magic trick: incremental zfs send with snapshots is symmetric. Once you've synchronized two copies this way you can do it in either direction and you can reverse directions partway through. My home directory's move is almost certainly temporary (and it's certainly temporary if problems arise) and as long as I retain the filesystems and snapshots on both sides I can move back just as fast as I moved in the first place. I don't have to make a full copy and then synchronize it and so on; I can just make a new snapshot on the new fileserver and send back the difference between my last 'move over' snapshot and this first 'move back' snapshot. Speaking as someone who's currently basically jumping up and down on a new fileserver that should be good but hasn't been fully proven yet, that's a rather reassuring thing.

(In fact if I'm cautious I can update my old home directory every night or just periodically, so that I won't lose much even if the new environment goes down in total flames somehow.)

PS: Since I've now tested this in both directions, I can say that you can zfs send a filesystem from Solaris 10 update 8 to OmniOS and then zfs send snapshots from it from OmniOS back to S10U8 (provided that you don't 'zfs upgrade' the filesystem while it's on OmniOS; if you do, it'll become incompatible with S10U8's old ZFS version (or at least so the documentation says, I haven't tested this personally for the obvious reasons)).

Sidebar: how we freeze filesystems when moving them

The magic sequence we use is to unshare the filesystem (since all of our access to filesystems is over NFS), set readonly=on, and then make the final move snapshot. After the newly moved filesystem is up and running we set canmount=off on the old version and then let it sit until we have a backup of the moved one and everything is known to be good and so on.

We almost always do zfs receive into unmounted filesystems (ie 'zfs receive -u ...'.)

Written on 19 March 2014.
« Simple versus complex marshalling in Python (and benchmarks)
Killing (almost) all processes on Linux is not recoverable »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Wed Mar 19 00:12:24 2014
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.