Why we're almost certainly staying with ZFS
A commentator on my last entry asked:
i think the implicit ZFS requirement is far too constraining. have you considered giving it up?
The short answer is no. But the long answer is more interesting.
ZFS has a number of advantages, but as far as I'm concerned the big feature for us is ZFS's flexible management of space. How we use ZFS here (also) is that professors or groups buy space in fixed chunks and then create however many filesystems (in whatever configuration) they want to use. As is standard in ZFS, these filesystems draw space from the pool as needed, with currently unneeded space free in the pool for any filesystem to use. From my perspective, the really important thing about this setup is that professors don't have to decide how much space a filesystem should get; generally their entire interaction with the system is to buy more space every so often.
Moving to anything that doesn't offer this flexible management of space is essentially a non-starter; the only situation where we'd even consider regressing would be if there was some problem that made ZFS impossible to use.
Moving to another storage system would be a big pain (as well as changing from something we're very familiar with to something more or less unproven). We'd have to develop an entirely new fileserver environment complete with new management tools and then migrate all of our data to the new environment; this involves a lot of work and user-visible downtimes (although they aren't global downtimes since we'd migrate filesystem by filesystem). Given this pain and that staying with ZFS is an option, it's not enough for any potential ZFS replacement to be just as good as ZFS; we really need to see some sort of clear improvement to make the pain worth it. The more clear improvement there is, the easier it is to justify a move.
('Doesn't require Solaris' is not a clear improvement here, partly because we can get that and still keep ZFS by going with FreeBSD.)
Right now, I see nothing production-ready that is even at parity with ZFS in terms of this sort of flexible space management. The closest is Linux's btrfs, but btrfs is not production ready as I define it (and it has only feature parity here as far as I know, and while I'd like to run Linux instead of Solaris it's not a big enough gain). Thus I can't see any alternatives that would even vaguely start persuading us to give up ZFS.
(It's worth noting that our implicit requirements are not just ZFS, they're ZFS, good NFS fileservice, and a solid iSCSI client implementation. We're no more interested in changing the NFS and iSCSI aspect of the current fileservers than we are in changing away from ZFS.)