ZFS's zfs receive has no error recovery and what that implies

May 7, 2017

Generally, 'zfs send' and 'zfs receive' are a great thing about ZFS. They give you easy, efficient, and reliable replication of ZFS filesystems, which is good for both backups and transfers (and testing). The combination is among my favorite ways of shuffling filesystems around, right up there with dump and restore and ahead of rsync. But there is an important caveat about zfs receive that deserves to be more widely known, and that is that zfs receive makes no attempt to recover from a partially damaged stream of data.

Formats and programs like tar, restore, and so on generally have provisions for attempting to skip over damaged sections of the stream of data they're supposed to restore. Sometimes this is explicitly designed into the stream format; other times the programs just use heuristics to try to re-synchronize a damaged stream at the next recognizable object. All of these have the goal of letting you recover at least something from a stream that's been stored and damaged in that storage, whether it's a tarball that's experienced some problems on disk or a dump image written to tape and now the tape has a glitch.

zfs receive does not do any of this. If your stream is damaged in any way, zfs receive aborts completely. In order to recover anything at all from the stream, it must be perfect and completely undamaged. This is generally okay if you're directly feeding a 'zfs send' to 'zfs receive'; an error means something bad is going on, and you can retry the send immediately. This is not good if you are storing the 'zfs send' output in a file before using 'zfs receive'; any damage or problem to the file, and it's completely unrecoverable and the file is useless. If the 'zfs send' file is your only copy of the data, the data is completely lost.

There are some situations where this lack of resilience for saved send streams is merely annoying. If you're writing the 'zfs send' output to a file on media as a way of transporting it around and the file gets damaged, you can always redo the whole process (just as you could redo a 'zfs send | zfs receive' pipeline). But in other situations this is actively dangerous, for example if the file is the only form of backup you have or the only copy of the data. Given this issue I now strongly discourage people from storing 'zfs send' output unless they're sure they know what they're doing and they can recover from restore failures.

(See eg this discussion or this one.)

I doubt that ZFS will ever change this behavior, partly because it would probably require a change in the format of the send/receive data stream. My impression is that ZFS people have never considered it a good idea to store 'zfs send' streams, even if they never said this very strongly in things like documentation.

(You also wouldn't want continuing to be the default behavior for 'zfs receive', at least in normal use. If you have a corrupt stream and you can retry it, you definitely want to.)

Comments on this page:

By cks at 2017-05-08 16:32:31:

My understanding is that resumable 'zfs send' and 'zfs receive' only help if you can get an intact send stream somehow. This is fine for interruptions (where you can resume from where things stopped), but doesn't help if you have genuine corruption that you would have to skip over.

By Anon at 2017-05-08 17:29:58:

This is fine for interruptions [...] but doesn't help if you have genuine corruption that you would have to skip over.

Does https://github.com/illumos/illumos-gate/commit/9c3fd1216fa7fb02cfbc78a2518a686d54b48ab8#diff-5d452ee490fd0347170ea2a477f43962R22 show the sort of corruption you were thinking of?

By cks at 2017-05-08 21:54:08:

Without reading the changeset in detail, it's at least a partial example of damage to a send stream. I think that a full version would somehow force an IO read error at some point as well (to simulate things like a ZFS filesystem that's detected a corrupt block and refuses to give it to the 'zfs receive').

Detecting the damage and reporting where is only part of the problem, though. The troublesome bit, on both a technical and 'how it should work' level, is how to recover from it and maximizing the amount that you can recover.

By Anon at 2017-05-10 15:24:39:

Hmm... I guess if you want to grab as much as you can regardless of "corruption" you could use the option mentioned in https://github.com/zfsonlinux/zfs/issues/1982 on the ZFS module itself? However, I admit this is getting a away from the whole automatic error recovery idea though.

By cks at 2017-05-10 16:57:53:

Unfortunately, as far as I understand zfs_send_corrupt_data is for the case where you have corrupted data on the snapshot source and you still want to be able to 'zfs send' a snapshot to your destination. I don't believe it helps you if the send stream itself becomes corrupt while you have it stored on disk or the like.

Written on 07 May 2017.
« A mistake I made when setting up my ZFS SSD pool on my home machine
Some things I've decided to do to improve my breaks and vacations »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun May 7 19:36:26 2017
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.