Wandering Thoughts archives

Why we're almost certainly staying with ZFS

A commentator on my last entry asked:

i think the implicit ZFS requirement is far too constraining. have you considered giving it up?

The short answer is no. But the long answer is more interesting.

ZFS has a number of advantages, but as far as I'm concerned the big feature for us is ZFS's flexible management of space. How we use ZFS here (also) is that professors or groups buy space in fixed chunks and then create however many filesystems (in whatever configuration) they want to use. As is standard in ZFS, these filesystems draw space from the pool as needed, with currently unneeded space free in the pool for any filesystem to use. From my perspective, the really important thing about this setup is that professors don't have to decide how much space a filesystem should get; generally their entire interaction with the system is to buy more space every so often.

Moving to anything that doesn't offer this flexible management of space is essentially a non-starter; the only situation where we'd even consider regressing would be if there was some problem that made ZFS impossible to use.

Moving to another storage system would be a big pain (as well as changing from something we're very familiar with to something more or less unproven). We'd have to develop an entirely new fileserver environment complete with new management tools and then migrate all of our data to the new environment; this involves a lot of work and user-visible downtimes (although they aren't global downtimes since we'd migrate filesystem by filesystem). Given this pain and that staying with ZFS is an option, it's not enough for any potential ZFS replacement to be just as good as ZFS; we really need to see some sort of clear improvement to make the pain worth it. The more clear improvement there is, the easier it is to justify a move.

('Doesn't require Solaris' is not a clear improvement here, partly because we can get that and still keep ZFS by going with FreeBSD.)

Right now, I see nothing production-ready that is even at parity with ZFS in terms of this sort of flexible space management. The closest is Linux's btrfs, but btrfs is not production ready as I define it (and it has only feature parity here as far as I know, and while I'd like to run Linux instead of Solaris it's not a big enough gain). Thus I can't see any alternatives that would even vaguely start persuading us to give up ZFS.

(It's worth noting that our implicit requirements are not just ZFS, they're ZFS, good NFS fileservice, and a solid iSCSI client implementation. We're no more interested in changing the NFS and iSCSI aspect of the current fileservers than we are in changing away from ZFS.)

What we (probably) want in a future version of Solaris

We're starting to be vaguely in sight of the point at where we'll have to consider alternatives to our current fileservers. While the software environment hasn't changed, our fileserver infrastructure is built on SunFire X2200s and X2100s and we've already had two hardware failures in our X2100 pool. We have some number of spare X2200s and of course there's more modern hardware, but the problem is Solaris licensing; there's a strictly limited pool of spare X2200s that we can legally use with Solaris.

Oracle Solaris (on new hardware) is not an option. It costs too much (especially once you include likely hardware costs), we'd have no source code access (which has been very important to us), and most of all I have no trust in Oracle's long-term behavior. If we're building a new fileserver environment that's supposed to last at least five years, Oracle's handling of Solaris and Solaris licensing issues to date has created too much uncertainty for me.

The straightforward alternative version of 'Solaris' is one of the Illumos-based distributions. This raises the question of 'which one', which in turn (for me) raises the question of just what we want and need in the 'Solaris' that we use.

I'm afraid that my answer is going to be boring: we want a traditional Unix server OS. We want something with an /etc/passwd, cron, et al, something that we log in to through SSH and get a Bourne shell command line with vi and so on. We're going to continue to configure and manage ZFS pools our way and with our own tools (and we're going to continue to use iSCSI in a very specific way); we have no interest in web management interfaces, integrated NAS systems, and so on. I'd like to see better package management and more packages in the base system (it's 2012, everyone should be packaging rsync), but if I'm being honest it's not a very high priority because we're not going to update these systems unless we can't help it.

We absolutely need source code. Lack of source code almost certainly would rule an Illumos-derived distribution out of the running entirely. Among other reasons for source, DTrace is important and it's not very useful without source.

Support is not a priority. Having support nominally available is reassuring but in practice we've never gotten actual support from Sun and it's all but certain that source code and open source support channels (IRC, etc) will provide better results. Similarly, having the specific release of a distribution updated for a long time is also reassuring but probably not useful in practice; we're extremely likely to treat these machines as appliances and never apply updates to them.

(If nothing else, having no official support available will waste less time because I won't be spending a bunch of time talking to a support organization that doesn't actually solve my problems.)