Why we wound up using Linux for our iSCSI targets

January 8, 2013

I was recently asked in email why we chose Linux for our iSCSI targets and if it had to do with Linux's wide variety of hardware support. The answer is 'sort of', but there's a story here (which I'm going to simplify a bit).

To start with, our fileserver environment didn't appear out of nowhere; instead it's an evolution of an earlier fileserver environment, one that was built out of SPARC Solaris NFS servers talking to (mostly SATA) RAID appliances over (lower-end) FibreChannel. When we set out to modernize this but keep the same basic shape, our initial decision was to stay with an 'appliance' style solution for the backend disks rather than try to engineer our own backends. Doing our own backends might wind up cheaper, if they worked and if you didn't count staff time, but it exposed us to a bunch of risks that didn't seem worth it for what looked like relatively modest savings (at our planned size). So we picked Solaris 10 on x86 as the NFS fileserver, iSCSI over 1GB Ethernet as the SAN interconnect, and found a nice iSCSI appliance.

Then the iSCSI appliance failed badly. This sank plan A in a rather spectacular and grim fashion, and (along with some other things) left us feeling not really happy about appliance solutions in general. Suddenly it seemed worthwhile to try building our own test iSCSI backend to see how it would work (we had plenty of random spare servers, although I think we may have bought one for testing).

At the time we did this, in late 2007 and early 2008, our choices for the OS on a build-our-own iSCSI target boiled down to Linux and Solaris. There were two problems with Solaris. First, at the time the Solaris iSCSI target implementation simply wasn't mature (eg). Second and less important, it looked very much like the cheap ways to connect a dozen or so SATA disks to a machine would involve hardware that had Linux drivers and did not have Solaris drivers that I could see. By contrast the Linux iSCSI target worked (and performed well), we were familiar with Linux already, and Linux supported all the hardware we needed. By the time the dust settled there wasn't much of a choice.

Linux was a risk in some ways because our iSCSI target software was third-party software, although open source and with active developers. It would have been more reassuring to go with an integrated, fully-supported iSCSI target implementation, but we didn't have that option. What we could get on Linux did work, so we went with it.

(It's possible that we overlooked an iSCSI target implementation on FreeBSD at the time. I'm not sure that it would have made a difference unless the FreeBSD version walked on water, because we had no other experience with FreeBSD. When in doubt we pick the OS that we already run other instances of.)

Written on 08 January 2013.
« Why bad support matters (war story included)
It turns out I'm biased towards kernel iSCSI target implementations »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Tue Jan 8 00:42:51 2013
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.