Chris's Wiki :: blog/solaris/ZFSWhyStaying

Why we're almost certainly staying with ZFS

December 24, 2012

A commentator on my last entry asked:

i think the implicit ZFS requirement is far too constraining. have you considered giving it up?

The short answer is no. But the long answer is more interesting.

ZFS has a number of advantages, but as far as I'm concerned the big feature for us is ZFS's flexible management of space. How we use ZFS here (also) is that professors or groups buy space in fixed chunks and then create however many filesystems (in whatever configuration) they want to use. As is standard in ZFS, these filesystems draw space from the pool as needed, with currently unneeded space free in the pool for any filesystem to use. From my perspective, the really important thing about this setup is that professors don't have to decide how much space a filesystem should get; generally their entire interaction with the system is to buy more space every so often.

Moving to anything that doesn't offer this flexible management of space is essentially a non-starter; the only situation where we'd even consider regressing would be if there was some problem that made ZFS impossible to use.

Moving to another storage system would be a big pain (as well as changing from something we're very familiar with to something more or less unproven). We'd have to develop an entirely new fileserver environment complete with new management tools and then migrate all of our data to the new environment; this involves a lot of work and user-visible downtimes (although they aren't global downtimes since we'd migrate filesystem by filesystem). Given this pain and that staying with ZFS is an option, it's not enough for any potential ZFS replacement to be just as good as ZFS; we really need to see some sort of clear improvement to make the pain worth it. The more clear improvement there is, the easier it is to justify a move.

('Doesn't require Solaris' is not a clear improvement here, partly because we can get that and still keep ZFS by going with FreeBSD.)

Right now, I see nothing production-ready that is even at parity with ZFS in terms of this sort of flexible space management. The closest is Linux's btrfs, but btrfs is not production ready as I define it (and it has only feature parity here as far as I know, and while I'd like to run Linux instead of Solaris it's not a big enough gain). Thus I can't see any alternatives that would even vaguely start persuading us to give up ZFS.

(It's worth noting that our implicit requirements are not just ZFS, they're ZFS, good NFS fileservice, and a solid iSCSI client implementation. We're no more interested in changing the NFS and iSCSI aspect of the current fileservers than we are in changing away from ZFS.)

Comments on this page:

From 68.195.89.131 at 2012-12-24 11:27:39:

thanks for the detailed response to my original comment. i'm still hungry for details, though. what does ZFS offer over the combination of LVM+ext4? note that often a disruptive technology offers fewer features because other factors (cost, capacity) dominate the equation. also how does iSCSI fit in your picture?

i've built similar systems (perhaps for a smaller number of 'customers') out of purely commodity hardware & linux. as a rule of thumb, compared to solaris or big storage company solutions, commodity solutions offered twice the capacity for half the price. these setups weren't perfect. CIFS, in particular, was always a source of problems (lockups).

/mark kennedy mtk@acm.org

By cks at 2012-12-24 22:36:01:

To the best of my knowledge, LVM+ext4 still requires manual intervention to resize filesystems so that they have available more space or use less space. This is fundamentally different from ZFS's flexible and automatic space management (in ZFS, filesystems are more or less an accounting mechanism).

For a description of our fileserver environment, including how we use iSCSI, start at ZFSFileserverSetup and follow appropriate links.

I don't believe that we could build an equivalent system without Solaris (even with feature parity in a ZFS replacement) for any noticeably smaller amount of money. Our Solaris fileservers use the same sort of inexpensive commodity hardware that we would (and do) run Linux on, so the only extra expense is Solaris licenses and those were very low when we got them.

(Oracle Solaris licenses are now much more expensive, which is one reason we are not looking at Oracle Solaris 11.)

From 91.13.157.7 at 2012-12-26 09:57:12:

I have been impressed with the stability of ZFSonLinux recently; while my workload is nowhere near yours, it felt less problematic than ZFS on FreeBSD, even...

By cks at 2012-12-27 02:03:33:

I was going to say that I have a reflexive distrust of ZFS on Linux efforts, but it turns out that I'm somewhat less distrustful than I thought I would be once I actually looked at the main one. I still feel there's enough questions hanging over such a project to steer me away from it, though.

(But it's now probably something we'll wind up evaluating, though. At least if it stabilizes in time.)

By trs80 at 2012-12-28 08:26:49:

You are missing one alternative, which is NetApp Data ONTAP. I assume it's implicitly eliminated due to the cost and the fact it's not open source, but apart from that it is quite nice.

By cks at 2012-12-28 14:28:43:

Cost is the major factor in eliminating things like NetApp. The units are expensive, the disks that go in them are expensive, the stuff like backup software is expensive, and so on. Unfortunately we just don't have that sort of money for infrastructure.

(Open source is not a major consideration by itself. The problem with Solaris 11 and lack of source is that we know that we need source code to make various bits of Solaris work for us.)

Written on 24 December 2012.

«	What we (probably) want in a future version of Solaris

A confession: Python 3 has always made me kind of angry