Thinking about FreeBSD versus Illumos for our ZFS fileservers
One of our options as a replacement for Solaris 10 is to switch from Solaris to FreeBSD (now that I no longer believe you need Solaris for ZFS). There are both advantages and uncertainties to such a move and today I want to ramble on a bit about how I see both sides.
On the one hand, we have no particular attraction to Solaris as Solaris. On the other hand, Solaris is the devil we know and this particular devil works pretty well for us (plus I've come around to the idea that DTrace is kind of useful). Using some version of Illumos preserves what we have now while allowing us to use new hardware; we'd also get various improvements in things like package management.
There are two or three drawbacks of Illumos: picking the right distribution, the long term development path, and potentially hardware support. The right distribution is not just a matter of what is good today; given that we'll likely be running these machines for five years or more, we care about the long term viability of the distribution and ideally the continued availability of security updates for old versions. The question of long term development in general is, well, is Illumos going to survive five years or more in useful form (ie, something that will run on a generic server as a generic server OS) or is it going to wither away into at best a narrowly specialized thing? It takes a bunch of work to keep developing a general server OS and there are already lots of other things to drain away potential contributors.
I had to talk about Illumos's drawbacks because FreeBSD is the flipside of them. With FreeBSD we get what is for us a new and untested platform but it has a lot of momentum and history behind it, the policies it operates under are clear, and it seems clearly supported well into the future (since FreeBSD is basically the non-Linux Unix). FreeBSD also has the advantage of dropping a lot of things about Solaris that I don't like in favour of a bunch of well-proven technology and plain modern stuff that I am much happier with.
In short, FreeBSD gives us a more attractive overall system at the cost of some uncertainty over the pieces that we really care about. It also seems likely that FreeBSD DTrace is less mature than Illumos DTrace and as mentioned I sort of care about DTrace these days. Of course this uncertainty can be somewhat mitigated with testing and other people's experiences.
(It's not just an issue of things like normal functionality and performance, although those matter. We also care about the dark corners, which you can't test and you sort of have to take on either trust or painful experience. Illumos lets us give more weight to some of our painful Solaris experience, since it's likely that Illumos is going to be very Solaris-like in many ways.)
PS: I consider it a feature that FreeBSD can be installed with non-ZFS root filesystems. Given some of the ZFS failure modes I've seen I actively prefer not to need ZFS to boot the machine. I'm not sure how many Illumos derived distributions still support this (if any); my impression was that the Solaris world was going full steam ahead to an all-ZFS future.