ZFS changes filesystem device numbers on reboot
Here is an unpleasant discovery that we just finished making: in at least Solaris 10 U6, ZFS changes the device numbers of all of your filesystems when your system reboots. More specifically (and even worse), the device numbers of a pool's filesystems change whenever the pool is imported, including the implicit import that happens when a system boots up.
(This raises interesting questions of how NFS filehandles keep working across system reboots, which they do. Presumably they have some other stable identifier for a given ZFS filesystem that still fits in the NFS filehandle size limits.)
So, you might ask, what difference does this make? Unfortunately for ZFS (and people using it), some programs still count on device numbers being stable so that they can use the device number plus inode as a long term identifier for a particular file. One such program is GNU tar, which uses this assumption for its incremental backups. If the assumption is violated, GNU tar's incremental backups turn into complete 'level 0' backups, since they think that everything is new because no device numbers match.
So, if you have ZFS machines and a backup system that might possibly assume that device numbers are stable, be careful. If you haven't tested rebooting a machine with enough backed-up data that you can tell if you're suddenly doing a full backup instead of an incremental one, well, you might want to plan such a test.
(For GNU tar specifically it is possible to fix the situation by directly modifying GNU tar's index files, which are fortunately stored in plain text format. How to do so doesn't fit within the margins of this entry.)