Thinking about FreeBSD versus Illumos for our ZFS fileservers

January 30, 2013

One of our options as a replacement for Solaris 10 is to switch from Solaris to FreeBSD (now that I no longer believe you need Solaris for ZFS). There are both advantages and uncertainties to such a move and today I want to ramble on a bit about how I see both sides.

On the one hand, we have no particular attraction to Solaris as Solaris. On the other hand, Solaris is the devil we know and this particular devil works pretty well for us (plus I've come around to the idea that DTrace is kind of useful). Using some version of Illumos preserves what we have now while allowing us to use new hardware; we'd also get various improvements in things like package management.

There are two or three drawbacks of Illumos: picking the right distribution, the long term development path, and potentially hardware support. The right distribution is not just a matter of what is good today; given that we'll likely be running these machines for five years or more, we care about the long term viability of the distribution and ideally the continued availability of security updates for old versions. The question of long term development in general is, well, is Illumos going to survive five years or more in useful form (ie, something that will run on a generic server as a generic server OS) or is it going to wither away into at best a narrowly specialized thing? It takes a bunch of work to keep developing a general server OS and there are already lots of other things to drain away potential contributors.

I had to talk about Illumos's drawbacks because FreeBSD is the flipside of them. With FreeBSD we get what is for us a new and untested platform but it has a lot of momentum and history behind it, the policies it operates under are clear, and it seems clearly supported well into the future (since FreeBSD is basically the non-Linux Unix). FreeBSD also has the advantage of dropping a lot of things about Solaris that I don't like in favour of a bunch of well-proven technology and plain modern stuff that I am much happier with.

In short, FreeBSD gives us a more attractive overall system at the cost of some uncertainty over the pieces that we really care about. It also seems likely that FreeBSD DTrace is less mature than Illumos DTrace and as mentioned I sort of care about DTrace these days. Of course this uncertainty can be somewhat mitigated with testing and other people's experiences.

(It's not just an issue of things like normal functionality and performance, although those matter. We also care about the dark corners, which you can't test and you sort of have to take on either trust or painful experience. Illumos lets us give more weight to some of our painful Solaris experience, since it's likely that Illumos is going to be very Solaris-like in many ways.)

PS: I consider it a feature that FreeBSD can be installed with non-ZFS root filesystems. Given some of the ZFS failure modes I've seen I actively prefer not to need ZFS to boot the machine. I'm not sure how many Illumos derived distributions still support this (if any); my impression was that the Solaris world was going full steam ahead to an all-ZFS future.

Comments on this page:

From at 2013-01-30 08:16:55:

Here is a excellent site on building a ZFS fileserver, with lots of picture and benchmarks.

On the latest posted test results, FreeNAS was able to keep up with Nexenta.

I would go along with your think, FreeBSD has been around for a long time and it doesn't seem to be going away anytime soon.


Cheers, Paul

From at 2013-01-30 08:21:58:

One more thing....

Non ZFS boot can be a plus, yet I like the ability to rollback updates you may have applied to your OS. Nobody like a bad OS update/upgrade!

ZFS is complicated enough already and you don't want to deal with any potential ZFS complications on a boot volume.

Cheers, Paul

From at 2013-01-30 20:29:08:

The biggest problem I have with FreeBSD is the 1 year (at best) support cycle.

From at 2013-01-31 07:44:29:

It's not true that FreeBSD only offers a 1 year support cycle. Instead they provide releases with an extended lifetime of 2 years. See

From at 2013-01-31 10:58:28:

We had the same issue at my place of business, the same concerns, and ultimately ended up going with FreeBSD 9 as the future ZFS platform of choice due to exactly the line of reasoning you mentioned. Samba has gotten good enough with ZFS where, with minimal messing around, you can have a file server with CIFS permissions that can be managed from a Windows box the same as any other Windows file server. Performance is great (but not as good as the kernel CIFS server in OpenSolaris was), and NFS is obviously rock-solid and very performant on FreeBSD as well.

The one thing that I really miss from OpenSolaris that FreeBSD can't hold a candle to is the simplicity and robustness of the iSCSI stack. FreeBSD's doesn't even come close.

From at 2013-02-01 08:18:53:

You shouldn't necessarily draw your conclusions in regards to Illumos or Solaris 11 based on your experiences with Solaris 10. S10 is very old and much behind S11 and Illumos, especially when it comes to ZFS.

Currently there isn't a general OS distro based on Illumos which is being actively developed. Even Illumos itself is not really that actively being developed from that point of view - there is much progress there but only in areas where it is being comercially being used (Joyent, Nexenta, Delphix) - everything else is stagnated. Right now it still doesn't look that bad but it will.

On the other hand if all you need is a file server and not a general purpose OS, then Nexenta might be an option.

Now, if you really care and value your data I would suggest to look at Solaris 11. You can run it on 3rd party x86 hardware with full support for relatively little money - $1k per CPU socket per year. For file servers a modern 2-socket server is going to be an overkill, so Solaris would cost you $2k per year per server - this is not really that much. This would give you peace of mind for the next 5 years you mentioned, commercial support, access to security updates and bug fixes, etc. And this would give you a general OS which is being actively developed and bug fixed in many different areas Illumos isn't.

I know it is "cool" to bash Oracle these days, but if you look at it calmly and from distance, then it well might make sense to go with Solaris 11.

Unless your data is not really worth to you $1-2k per year and then you are willing to run it on other platforms. In this case FreeBSD is fine choice.

By cks at 2013-03-04 12:40:58:

Department of slow reactions:

I strongly disagree with the previous comment. My replies are sufficiently long that they've wound up in two entries, The strikes against Solaris 11 for us and the mythology of spending money on things.

Written on 30 January 2013.
« Some patterns for easy to parse Domain Specific Languages in Python
Why you should support specifying comparison keys (and functions) »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Wed Jan 30 00:20:35 2013
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.