The strikes against Solaris 11 for us
A commentator on my entry thinking about FreeBSD for future ZFS-based fileservers left a comment that contains any number of things that I want to react to.
You shouldn't necessarily draw your conclusions in regards to Illumos or Solaris 11 based on your experiences with Solaris 10. S10 is very old and much behind S11 and Illumos, especially when it comes to ZFS.
I've written about our view on ZFS features before, although focused mostly on later versions of Solaris 10. The quick version is that I still can't see any new ZFS features that are especially enticing to us. It is vaguely possible that Solaris 11 ZFS contains bug fixes for issues that we might encounter in the future, but we certainly encountering any serious issues today so this is not very compelling. Not when set against the other costs.
Since this is long, I am going to give you my summary view of the other costs up front. They are no Solaris source code, that we have to trust Oracle to keep a licensing model I don't think they're very enthused about, and that it costs anywhere between $7k/year and $20k/year and up (depending on just what fileservers we wind up with).
(For that matter, we have to trust Oracle to keep going with Solaris at all. I am far from convinced about this; Oracle is relatively ruthless about things that do not make them good money, good Solaris development is expensive, and I do not see how Solaris makes Oracle much money especially over the long term.)
The very first cost is that moving to Solaris 11 means no more source code access because Solaris 11 is closed source. This is a very big issue for us. No source code makes DTrace almost useless to us (cf) and DTrace was very important for solving a recent major performance issue. Our ZFS spares system also relies crucially on being able to extract and interpret non-public ZFS information (because we have no real choice; the information we need is not available through public interfaces).
Now, if you really care and value your data I would suggest to look at Solaris 11. You can run it on 3rd party x86 hardware with full support for relatively little money - $1k per CPU socket per year.
I have many reactions to this. One of them is that I completely reject the idea that Solaris 11 is the only right choice if we 'really care and value [our] data'. Paying money for something does not make it either good or better than the alternatives; if anything, my experience has been the exact opposite.
In addition there is a major issue here, which is that this approach requires extending a significant amount of trust to Oracle. What happens if in two years Oracle decides that this licensing scheme was a bad idea and withdraws it, effective immediately (or just significantly increases prices)? Don't say it can't or won't happen; Oracle has made similar abrupt changes in Solaris licensing before. My personal view is that this is especially likely to happen because a $1k per CPU socket price is not something that I think of as attractive. I don't think that Oracle actually wants people to use this program, which makes it especially likely to change or disappear and thus dangerous.
For file servers a modern 2-socket server is going to be an overkill, so Solaris would cost you $2k per year per server - this is not really that much.
One way to put my reaction to this statement is that it shows the vast gulf between (some sorts of) commercial businesses and an academic environment. In an academic environment such as mine, $2k/server/year is a very big sum of money; it is more than it would cost to replace the server outright every year. We do not have (at current server usage) even $7k/year to pay for Solaris 11 licenses, much less something like $20k/year (for ten dual-socket fileservers, if our environment expands).
The only way we could even start trying to justify and get $1k/year per server for software is if the software did something truly amazing and essential. Solaris 11 does not qualify. If our only options were to pay $1k/year per server or abandoning ZFS entirely, I'm pretty sure that we would be abandoning ZFS. Certainly in an argument between FreeBSD (free, we get source code, etc) and Solaris 11 ($7k+/year, no source code, etc), I do not think I could possibly successfully defend Solaris 11.
It's worth noting one subtle effect of a per-year, per-fileserver licensing cost: it makes expanding our environment much more expensive. In a non Solaris 11 world we could add more fileservers for just the hardware costs and those are relatively low (especially if we (re)use servers that we already have). If we licensed Solaris 11 any new fileserver would be a $1k-$2k/year cost over and above the raw hardware cost. This would probably mean no new fileservers.
This would give you peace of mind for the next 5 years you mentioned, commercial support, access to security updates and bug fixes, etc.
I will condense my reactions to this to just saying that our experiences with Sun on all of these measures has not been all that good and I have no reason to assume that Oracle will provide any better experiences than Sun did. In practice I assume that commercial vendors will provide us no support and perhaps some security updates, regardless of what they are nominally paid to do.
I have additional reactions to other bits of the comment but they are not really about Solaris 11 as such, so I think I will stop this entry here.