Wandering Thoughts archives

The future of OmniOS here if we can't get 10G-T working on it

When I wrote about our long road to getting 10G in production on OmniOS after our problems with it, I mentioned in an aside that the pessimistic version of when we might get our new fileserver environment back to 10G was 'never' and that that would have depressing consequences. Today I've decided to talk about them.

From the start, one of my concerns with Illumos has been hardware support. A failure to get our OmniOS fileservers back to 10G-T would almost certainly be a failure of hardware support, where either the ixgbe driver didn't get updated or the update didn't work well enough. It would also be specifically a failure to support 10G. Both have significant impacts on the future.

We can, I think, survive this generation of fileservers without 10G, although it will hurt (partly because it makes 10G much less useful in other parts of our infrastructure and partly because we spent a bunch of money on 10G hardware). I don't think we can survive the next generation without 10G; in four years 10G-T will likely be much more pervasive and I'm certainly hoping that big SSDs will be cheap enough that they'll become our primary storage. SSDs over 1G networking is, well, not really all that attractive; once you have SSD data rates, you really want better than 1G.

That basically means the next generation of fileservers could not be OmniOS (maybe unless we do something really crazy); we would have to move to something we felt would give us 10G and the good hardware support we hadn't gotten from Illumos. The possibility of going to a non-Illumos system in four years obviously drains off some amount of interest in investing lots of time in OmniOS now, because there would be relatively little long term payoff from that time. The more we think OmniOS is not going to be used in the next generation, the more we'd switch to running OmniOS purely in maintenance mode.

To some extent all of this kicks into play even if we can move OmniOS back to 10G but just not very fast. If it takes a year or two for OmniOS to get an ixgbe update, sure, it's nice to be running 10G-T for the remainder of the production lifetime of these fileservers, but it's not a good omen for the next generation because we'd certainly like more timely hardware support than that.

(And on bad omens for general hardware support, well, our version of OmnioOS doesn't even seem to support the 1G Broadcom ports on our Dell R210 servers.)

Sidebar: I'm not sure if Illumos needs more development in general

I was going to say that lagging hardware support could also be a bad omen for the pace of Illumos development in general, but I'm actually not sure if Illumos actually needs general development (from our perspective). Right now I'm assuming that the great 'ZFS block pointer rewrite' feature will never happen, and I'm honestly not sure if there's much other improvements we'd really care very much about. DTrace, the NFS server, and the iSCSI initiator do seem to work fine, I no longer expect ZFS to get any sort of API, and I don't think ZFS is missing any features that we care about very much (and we haven't particularly tripped over any bugs).

(ZFS is also the most likely thing to get further development attention and bugfixes, because it's currently one of the few big killer features of Illumos for many people.)

Our likely long road to working 10G-T on OmniOS

I wrote earlier about our problems with Intel 10G-T on our OmniOS fileservers and how we've had to fall back to 1G networking. Obviously we'd like to change that and go back to 10G-T. The obvious option was another sort of 10G-T chipset besides Intel's. Unfortunately, as far as we can see Intel's chipsets are the best supported option and eg Broadcom seems even less likely to work well (or at all, and we later had problems with even a Broadcom 1G chipset under OmniOS). So we've scratched that idea; at this point it's Intel or bust.

We really want to reproduce our issues outside of production. While we've set up a test environment and put load on it, we've so far been unable to make it fall over in any clearly networking related way (OmniOS did lock up once under extreme load, but that might not be related at all). We're going to have to keep trying in the new year; I don't know what we'll do if we can't reproduce things.

(We also aren't currently trying to reproduce the dual port card issue. We may switch to this at some point.)

As I said in the earlier entry, we no longer feel that we can trust the current OmniOS ixgbe driver in production. That means going back to production needs an updated driver. At the moment I don't think anyone in the Illumos community is actively working on this (which I can't blame them for), although I believe there's some interest in doing a driver update at some point.

It's possible that we could find some money to sponsor work on updating the ixgbe driver to the current upstream Intel version, and so get it done that way (assuming that this sort of work can be sponsored for what we can afford, which may be dubious). Unfortunately our constrained budget situation means that I can't argue very persuasively for exploring this until we have some confidence that the current upstream Intel driver would fix our issues. This is hard to get without at least some sort of reproduction of the problem.

(What this says to me is that I should start trying to match up driver versions and read driver changelogs. My guess is that the current Linux driver is basically what we'd get if the OmniOS driver was resynchronized, so I can also look at it for changes in the areas that I already know are problems, such as the 20msec stall while fondling the X540-AT2 ports.)

While I don't want to call it 'ideal', I would settle for a way to reproduce the dual card issue with simply artificial TCP network traffic. We could then change the server from OmniOS to an up to date Linux to see if the current Linux driver avoids the problem under the same load, then use this as evidence that commissioning an OmniOS driver update would get us something worthwhile.

None of this seems likely to be very fast. At this point, getting 10G-T back in six months seems extremely optimistic.

(The pessimistic view of when we might get our new fileserver environment back to 10G-T is obviously 'never'. That has its own long-term consequences that I don't want to think about right now.)

Sidebar: the crazy option

The crazy option is to try to learn enough about building and working on OmniOS so that I can build new ixgbe driver versions myself and so attempt either spot code modifications or my own hack testing on a larger scale driver resynchronization. While there is a part of me that finds this idea both nifty and attractive, my realistic side argues strongly that it would take far too much of my time for too little reward. Becoming a vaguely competent Illumos kernel coder doesn't seem like it's exactly going to be a small job, among other issues.

(But if there is an easy way to build new OmniOS kernel components, it'd be useful to learn at least that much. I've looked into this a bit but not very much.)