My experience with ZFS on Linux: it's worked pretty well for me

December 26, 2014

A while back I wrote an entry about the temptation to switch my office workstation to having my data in ZFS on Linux. Not that long after I wrote the entry, in early August according to 'zpool history', I gave in to the temptation and did just that. I've been running with everything except the root filesystem and swap in a ZoL pool ever since, and the short summary is everything has worked smoothly.

(Because I'm crazy, I did some wrestling with systemd and bind mounts instead of just using symlinks. That was about all of the hassles required to get things going.)

I'm running the development version of ZFS, hot from the git repository and generally the latest version, mostly because I wanted all of the bugfixes and improvements from the official release. At this point, I've repeatedly upgraded kernels (including over several significant kernel bumps) without any problems with the ZFS modules being able to rebuild themselves under DKMS. There was one unfortunate bit where the ZFS modules got broken by a DKMS update because they'd been doing something that was actually wrong, but that would have been much less alarming (ie, less of a surprise) if I'd actually paid attention to various messages and failures during my 'yum upgrade' run.

(The issue was also promptly fixed by the ZFS on Linux people, and I fixed it by reverting to the previous version of the Fedora DKMS package.)

My moderately big worry beforehand was memory issues, but I didn't really have any problems. I ran into one interesting issue with kernel memory usage that I'm going to write up as another entry, but it wasn't a problem as such; the short summary is that ZFS was using too little memory instead of too much. However my system did get happier with life when I increased its memory from an already relatively large 16 GB all the way up to 32 GB, and I don't know how ZFS on Linux would do on a machine with only, say, 4 GB of RAM.

The major benefit of using ZoL instead of my previous approach of ext4 over LVM over MD has been the great reassurance value of ZFS checksums and 'zpool scrub'. I took advantage of ZFS's support for L2ARC to safely add a caching SSD to the pool (which is not currently possible with Fedora's version of LVM), but I don't know if it's doing me much good. And of course I like the flexible space usage that ZFS gives me, as I no longer have to preallocate space to each filesystem and then watch them all to expand them as needed. I haven't attempted any sort of performance measurements, so all I say is that my machine doesn't feel noticeably slower or faster than before.

(I've taken advantage of ZFS snapshots a couple of times in little ways, but so far that's basically just been playing around with them on Linux.)

Thus, on the whole I would say that if you're tempted by the idea of ZoL that you should try it out, and you should definitely try it out from the latest development version. I watch the ZoL git changelogs carefully and the changes that get committed are almost entirely fixes for bugs, with periodic improvements and things from other ZFS implementations. Building RPMs for a DKMS install from the source tree is just as simple as the ZoL site describes it.

(I was hoping to be able to report that my ZoL setup had also survived a Fedora 20 to Fedora 21 upgrade without problems, but I haven't been able to try that for other reasons. I honestly don't expect any problems because the F20 and F21 kernel versions are just about the same. I did do a test upgrade on a virtual machine with the ZoL DKMS packages installed and a ZFS pool configured, and had no problems with it (apart from the systemd crash, which happens with or without ZoL).)

After this positive experience (and after reading about various btrfs things), I expect that my next home machine build will also use ZFS on Linux. ZFS checksums and pool scrubs are enough of a reason to do it all on their own, never mind the other advantages. But that's probably several years in the future at this point, as my current home machine isn't that old by my standards.

Comments on this page:

I have had the same kind of experience on my home machine and on my previous office workstation, and I'm quite confident ZoL won't eat my data any more than ext4.

In my previous job, I have also built a reasonably big archive server ( which had (and I guess still has) about half of its disk space as ZFS over iSCSI; it was quite a simple setup (no deduplication, compression or even snapshots AFAIR) but it never gave me any trouble.

I've been following your posts on ZFS with great interest, and I am thinking of using ZFS on my next home server setup just so I get some experience with it.

Would you mind explaining the partitions and pools that you have on your desktop machine?

The purist in me would like to go the whole hog and use ZFS for root, but the various tutorials explaining how to do this leave me thinking this is a step too far.

John Allen Bofferdange, Luxembourg

By James (trs80) at 2014-12-28 00:07:04:

Given your praise here, and problems with OnmiOS have you considered migrating your OmniOS frontends to Linux? Obviously this would be a big undertaking (everyone hates downtime), but you've said before you have enough spare hardware that logistically it doesn't seem too hard.

By cks at 2014-12-28 21:59:55:

The short answer is that it's one thing to run ZFS on Linux on my workstation and another thing entirely to run it in a critical production role. ZoL appears to be good enough for the former, but I'm far from assured right now that it's good enough for the latter; among other things, people still report bad memory exhaustion issues every so often on the ZoL mailing lists.

(Linux also has a number of unsolved issues as a NFS fileserver in our environment, such as client authentication.)

I expect this to change over time, but probably not fast enough that I and my co-workers would be happy to replace the current, 'working fine at 1G' generation of OmniOS fileservers with new, untested Linux ZoL fileservers just to get 10G. 10G is nice, but it's not currently anywhere near that critical.

By cks at 2015-01-05 12:31:31:

Belatedly: John Allen, I've written up my partitioning in ZFSOnLinuxDiskSetup. Like you, my current view is that using ZFS for my root filesystem is some combination of too daring and too much work. This is especially the case in Fedora if you ever want to take advantage of things like fedup, which require booting a custom built alternate root environment that would have to be hacked to support ZFS and so on. Other distributions may have similar issues depending on how they do things like release to release upgrades.

(My personal view is that I won't want to use ZFS for my root filesystems until ZFS is officially supported in whatever distribution I use, which is sadly likely to be 'never'.)

Written on 26 December 2014.
« DNSSec in the real world: my experience with DNSSec
My ZFS On Linux memory problem: competition from the page cache »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Dec 26 02:20:44 2014
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.