ZFS changes filesystem device numbers on reboot

August 28, 2009

Here is an unpleasant discovery that we just finished making: in at least Solaris 10 U6, ZFS changes the device numbers of all of your filesystems when your system reboots. More specifically (and even worse), the device numbers of a pool's filesystems change whenever the pool is imported, including the implicit import that happens when a system boots up.

(This raises interesting questions of how NFS filehandles keep working across system reboots, which they do. Presumably they have some other stable identifier for a given ZFS filesystem that still fits in the NFS filehandle size limits.)

So, you might ask, what difference does this make? Unfortunately for ZFS (and people using it), some programs still count on device numbers being stable so that they can use the device number plus inode as a long term identifier for a particular file. One such program is GNU tar, which uses this assumption for its incremental backups. If the assumption is violated, GNU tar's incremental backups turn into complete 'level 0' backups, since they think that everything is new because no device numbers match.

GNU tar, as it happens, is at the heart of our backup system. (Not that you have much choice with ZFS, since it has no dump equivalent and no, zfs send doesn't count.)

So, if you have ZFS machines and a backup system that might possibly assume that device numbers are stable, be careful. If you haven't tested rebooting a machine with enough backed-up data that you can tell if you're suddenly doing a full backup instead of an incremental one, well, you might want to plan such a test.

(For GNU tar specifically it is possible to fix the situation by directly modifying GNU tar's index files, which are fortunately stored in plain text format. How to do so doesn't fit within the margins of this entry.)

Written on 28 August 2009.
« What packages SystemTap requires on Ubuntu 8.04 (and others)
MUST versus SHOULD in your specifications »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Aug 28 00:39:10 2009
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.