Why I am not enthused about ZFS for my root filesystem
One of the changes in Oracle's recent Solaris 11 preview (and in OpenSolaris releases before it) is that your root filesystem must be a ZFS filesystem; it can no longer be a UFS filesystem. While I understand why Oracle did this, it is not a change that leaves me feeling very enthused.
The short version of why I do not like this is that previously, the
entire ZFS subsystem could go belly-up and your system could still boot.
You might think that ZFS going belly-up entirely should not happen, but
the problem there is
/etc/zfs/zpool.cache, the system ZFS cachefile. This is a binary file that can only be maintained by
ZFS tools, and it at least used to be possible for it to become corrupt.
When it became corrupt, your method of fixing it was, uh, to remove it
and start again by re-importing your pools.
This is generally possible if your actual Solaris system filesystem is on UFS (both the 'removing' bit and the 'starting again' bit). My strong impression is that this is much harder if your root filesystem is ZFS, because you have a little chicken and egg problem.
(In fact at one point the official solution for the devices involved in the root pool changing names was 'boot from a rescue environment'. Yes, really. In an enterprise operating system, with self-identifying filesystems and storage pools. I hope that this has changed since then.)
Possibly the rescue environment has a well-honed solution to this problem (one that gets your root pool back and the system booting to single-user mode so that you can fix everything else), or perhaps this doesn't happen any more. But frankly, Solaris 10 has not impressed me with its resilience in the face of various events, so I am not inclined to trust it here; I would much prefer the simpler, far better tested approach of a UFS root filesystem.