Why we don't use jumbo frames for iSCSI: a cautionary tale on testing
When we were initially designing our iSCSI SAN environment, we planned to use jumbo frames; it was just an obvious thing to do. Then we tried to find an inexpensive 16 or 24 port gigabit Ethernet switch that actually did jumbo frames completely correctly.
To skip to the punchline, we failed. Worse, in the process of failing it became obvious that jumbo frames were a dangerous swamp. (Since jumbo frames did not get us any real performance increase, we didn't then try expensive switches.)
The easiest switches to deal with were the ones that didn't support jumbo frames and said so. More annoying were switches that claimed to support jumbo frames but were lying about it. The worst and most dangerous switches were ones that supported jumbo frames, but badly, and the champion switch at this had a really creative bug.
This particular switch did jumbo frames fine, passing traffic and running just as fast and reliably as you'd expect; I was quite happy, because I thought I'd finally found a winner. Then I power cycled it for an unrelated reason, and immediately jumbo frame TCP performance dropped off a cliff, going from wire speed to under a megabyte a second (but it didn't break completely). The switch configuration claimed that jumbo frames were enabled; re-enabling them suddenly cured the performance issues, at least until the next power cycle.
(This was clearly a firmware bug, and in retrospect I can guess where it was, but it made the switch useless; we couldn't use a switch that would effectively kill our entire iSCSI infrastructure after a power failure until it was fixed by hand.)
My experience with this switch convinced me of two things. First, it was going to be hard to find what we wanted; we were going to spend a lot of time auditioning and testing a lot of switches for not very much gain. Second and more important, jumbo frames were dangerous because they made switches have obscure failure modes. We ran into the power cycle issue mostly through luck; what if the next jumbo-supporting switch had an equally obscure issue that we didn't stumble over in testing and that only blew up in our face after a few months (or years) in production?
(Note that this is one of the cases where hindsight makes everything look obvious. Of course you should test power cycling a switch, it's so obvious. Except that it's not, because a switch that loses part of its configuration over a power cycle should never have gotten into production in the first place.)
What this really points out is the difficulty of fully testing something of even moderate complexity (at least from the outside, in black box testing). Part of it is the sheer size of how much you need to test, but a large part of it is our natural habit of jumping to assumptions in order to simplify our lives, especially if those assumptions are the way things have to be in order to work right. You can defeat these assumptions and the resulting mental blind spots, but only if you work hard at it, and you will wind up with a lot of work, most of which won't get you anything.
(And really, how much time is it sensible to spend doing tests that just tell you that equipment works the way it's claimed to, down to the obscure corner cases? In most environment this is like many-nines reliability; after a while, you hit the point of rapidly diminishing returns and you have to start making assumptions.)