ZFS's issues with long term storage management
Unfortunately, one of ZFS's current issues is that it does not have any direct support for the sort of data migration you need to do for long term storage management. There are three possible approaches to this:
- direct migration support: not present.
- growing a pool with the new storage then shrinking it by removing the old storage: while ZFS can grow pools it can't shrink them.
- add the new storage as a mirror and then detach the old storage: you can't mirror raidz or raidz2 pools.
The way you are supposed to use ZFS is to feed it raw disks and make raidz or raidz2 pools with them, but this leaves you without good migration options. Replacing individual disks as if they had failed and been replaced is not good enough; it takes out your redundancy for the duration of the rebuild, plus you can never change the number of disks that the storage pool is using.
If you let your backend handle the RAID bits and feed ZFS logical disks instead of physical ones, you can fake migration through mirroring (since you can mirror and then unmirror disks that are not part of a raidz or raidz2 pool). But doing this gives up some of ZFS's performance and reliability advantages and you don't get RAID 6 unless your backend supports it, which not all do.
(Shrinking ZFS pools is theoretically a planned future feature, but I can't make plans that count on it until it actually gets implemented and appears in Solaris.)