2007-04-17
A disappointment with ZFS
According to the ZFS documentation for the current official Solaris 10 release (I don't know about the state of OpenSolaris builds, and they're not of interest to us anyways), you cannot make a storage pool that is a mirror of RAID-Z2 pools.
(You can indirectly construct a mirror of RAID-5 pools, by building the RAID-5 pools with the Solaris Volume Manager instead of ZFS itself.)
We want some form of RAID-6 instead of RAID-5 because RAID-6 leaves you less exposed to data loss than RAID-5 plus a hot spare for the same number of disks. The problem with RAID-5 is that on modern SATA disks, resyncing onto your hot spare after you lose a disk may take an appreciable amount of time; during this time, you're exposed to data loss if you lose a second drive. Since you're going to have the second parity drive anyways (in the form of a hot spare), you might as well have it live.
We want mirroring so we can mirror across two iSCSI controllers, so we can survive losing an entire controller without user downtime. (The price of iSCSI based SATA storage is low enough that we can actually afford to do this.)
And finally, we need ZFS to do the RAID-Z2 because our chosen iSCSI controllers don't have RAID-6, only RAID-5. Since we want mirroring on top of the RAID-Z2 from a single controller, this means ZFS has to provide it too. The inability to do it is disappointing; if we use ZFS, we will have to live with mirroring over controller-based RAID-5.
(Technically you might be able to get ZFS to do this by making two RAID-Z2 pools, making one big filesystem on each, making one big file on each filesystem, and then turning around and telling ZFS to make a mirrored storage pool using those two files. Somehow I don't think this is an officially supported configuration, though.)