Wandering Thoughts: Recent Entries For 2012/05/31

What OSes have succeeded or failed here

In light of my recent entries, people might wonder how we've fared with operating systems here and which specific OSes have succeeded or failed. The necessary disclaimer is that this is all from my personal perspective; my co-workers might have a somewhat different view of things.

OSes that we've actively done things with while I've been here:

• OpenBSD: clear success within its domain.

  OpenBSD is our automatic choice for anything to do with firewalls (PF just works) and pretty much the default choice for related networking things like VPN servers, routers, and so on. We have a number of OpenBSD machines running additional simple network-related services and they just go and go without problems.

• Ubuntu LTS: a success but I'm not enchanted with it.

  Ubuntu LTS is our default Linux and thus our default Unix. Its combination of a long 'support' period and a wide vendor-supplied package collection continue to be unmatched (and be what we need), but it has various flaws and rough edges that leave me not really happy with it. Ubuntu LTS is more something that I put up with than something I actually like.

• Solaris: failed, as discussed.

  (We continue to build new fileservers with Solaris to match the current ones.)

• RHEL/CentOS: failed in practice.

  The short summary is that RHEL is not enough better than Ubuntu LTS and it has worse package availability (from Red Hat; we don't entirely trust EPEL, since it's a third party source). We've used RHEL on a few servers but the minor improvements (if any) haven't really been worth running another Linux distribution; the servers could as well run Ubuntu LTS and simplify our lives. I keenly regret this because I like RHEL better than Ubuntu LTS, but I have to face reality here; it's not enough better (and it has its own issues even apart from package availability).

  Our iSCSI backends run RHEL 5 and in theory the long support period is a great advantage for this. In practice they're appliances and we never update them anyways, so it's not clear how much this matters. We might stay with RHEL or CentOS in this specific situation even if we were redoing them from scratch today, just in case.

  (We continue to build new iSCSI backends with RHEL 5 to match the current ones.)

• Windows: TILT. Not applicable in our environment.

  We have one Windows terminal server for a specific, narrow purpose; to give our non-Windows users a way to run Windows programs (especially the Office suite). There's no interest in using Windows servers for anything else; as far as servers go, we're a Unix shop.

Worth mentioning:

• FreeBSD: never been seriously considered, would likely fail.

  I don't think we've ever seriously looked at running FreeBSD based servers. FreeBSD kind of fall into the 'RHEL zone', where it's different but without any solidly compelling advantage we've thought of. I've used FreeBSD enough in another environment to feel that it's nothing particularly special as Unixes go.

  It's possible that we'll wind up using FreeBSD as our next generation ZFS platform, but even then I don't think it would spread beyond that unless it turns out to be unbelievably cool.

  (In general, unless Linux somehow gets real native ZFS support I expect that any next generation ZFS platform would end up like Solaris is for us today, ie used as a special purpose appliance and no more. The most I'm hoping for is that the next generation platform is more pleasant than our current Solaris one.)

OSes that were pretty much before my time here:

• Debian: failed due to being supplanted by Ubuntu LTS.

  The problem with Debian in practice is that the support period is too short unless Debian does releases only very slowly. Beyond that, my impression is that Debian wound up being considered an inferior version of Ubuntu LTS; the few Debian based servers we used to have were replaced with ones based on Ubuntu LTS when we focused on the latter.

  (I actually built a Debian-based server here at one point (before we focused on Ubuntu LTS) but it never made it into production.)

• Red Hat Linux (now Fedora): failed, I believe in part because the support period is too short. Supplanted by Ubuntu LTS.

I don't think we've ever really looked at other Linux distributions; in general it seems unlikely that they have enough of an advantage over Ubuntu LTS to be attractive.

None of the other commercial Unixes are even on the radar (nor any of the other *BSDs; if we're not considering FreeBSD, they're even further down the list). Mac OS X is not something we run on servers, although we have some Macs and Windows machines around as test clients (since our users need our services to work with both).

Thinking about why Solaris has failed here

First off, as I rambled yesterday, to say that Solaris has failed here is not to say that our use of Solaris has been a failure (or that our Solaris machines have been); our fileservers are stable and run (generally) without problems and work. But at the same time it's clear that Solaris has failed to catch on here. The only strong reason our fileservers run Solaris is ZFS, and we've not even running anything close to a current version of Solarism at that (cf); at this point our fileservers are less servers and more sealed black-box appliances. We've never had any interest in using Solaris for anything else and we're not very enthused about Solaris even for ZFS; to put it one way, we pretty much use ZFS despite Solaris not because of it.

(There's good alternatives to Solaris for ZFS nowadays, but as it is our fileservers work and switching just to get away from Solaris seems uncompelling. Or even insane.)

Recently I've been thinking about the question of why Solaris failed here. At one level (as Solaris fans will tell you) Solaris is not unattractive; it has a number of nice features, including ZFS. But it clearly did not hit it off with us. Why? I think there are two major reasons.

The first is that Solaris is, frankly, somewhat erratic, flaky and buggy; it's not the kind of simple and solid system that, say, OpenBSD is. OpenBSD doesn't do very much but what it does just works and works reliably, which makes it a pleasure to use within its areas. Our Solaris machines are pretty stable but getting them there took a significant amount of work and local invention, and the reason they're stable is that we don't change anything on them and we don't ask them to do things that we know are problematic. Since Solaris has problems where we have looked, it probably also has problems where we haven't had to look yet.

The second is that Solaris doesn't have the package availability of many Linux distributions or FreeBSD, where you have a wide selection of relatively current open source software that is maintained by the vendor. There are outside efforts to provide open source packages for Solaris (and we use one of them), but outside efforts can disappear (and are not official) and so are intrinsically less trustworthy and confidence inducing than something the vendor itself supports. Unless Ubuntu absolutely explodes, a relatively current Apache is always only going to be an apt-get away and someone else will always handle security fixes for it. This is not something we could ever say about Solaris.

(Part of the problem of third-party packaging is that Solaris itself ships with any number of very old versions of things as official packages. All of the resulting options are bad ones.)

A significant contributing factor is that Solaris is simply not that pleasant to administer, partly because it is almost totally without modern niceties and partly because what it has in the way of administrative tools are often terribly bad (try to tell me with a straight face that Solaris 10's patching system is at all tolerable). Solaris administration is full of things that might have been good ideas if they were competently implemented but, well, they weren't.

(I understand that some of this has changed in modern Solaris, but by this point that's too little, too late (for us).)

The combination of these attributes in a single OS means there's nothing here to really feel enthused about using Solaris for. It lacks a speciality and a niche that it can really dominate in the way that OpenBSD dominates firewalls and related networking stuff, and it lacks the wide package availability (and administrative convenience) that would make it easy for us to deploy a Solaris machine to do <X> for some <X> like 'provide IMAP'.

(Our directly user accessible machines are essentially forced to run Linux (and Ubuntu at that), but we have plenty of servers that users don't log in to and that could theoretically run almost anything.)

As always, everyone's circumstances are different. For instance, if you build your entire software stack from source yourself in order to maintain full control over it you don't care in the least about Solaris's lack of prepackaged open source software; if it existed, you wouldn't use it anyways.