At this point we've finalized almost all of the hardware for renewing the hardware for our fileserver infrastructure, unless something terribly bad turns up (which is always possible). So today I feel like talking about what hardware we're choosing and the size and scope of our project, partly because it seems uncommon to do this sort of thing.

The base backend hardware is a SuperMicro X9SRH-7TF motherboard with 8G of (ECC) RAM, an Intel E5-2603 CPU, an extra LSI 9207-8i SAS controller card, and some additional networking. This gives us a single-socket motherboard with dual Intel 10G-T ports, which is the important thing for us because it makes 10G-T cheap enough that we can afford it. We need the LSI card because we're talking to SATA disks so we want to avoid SAS expanders and the motherboard only has 8 SAS ports onboard. All of this is in a SuperMicro SC 836BA-R920 case, which gives us 16 3.5" front panel drive bays for iSCSI data disks and two rear 2.5" drive bays for mirrored SSDs as the system disks.

(For backends the additional networking is likely to be a cheap Realtek 1G card. For fileservers it'll be a dual Intel 1G or 10G-T, depending on what we can afford. Fileservers will also have other hardware variance, such as a lot more memory and probably no LSI card.)

Our current plans for disks for backends is twelve 2TB WD Se drives (7200 RPM SATA drives with a five year warranty) plus four SSDs for ZFS ZILs; we haven't selected the SSDs yet. It's possible that we'll shift to one or two more HDs and less ZIL SSDs. The system SSDs will be a pair of semi-random 60 GB SSDs, since you don't need more than that for your system disks (well, you hardly need even that).

At the moment we have three primary HD-based fileservers with two backends each, one SSD-based fileserver with three backends, one further fileserver which now doesn't need to be on separate hardware, a hot spare backend (with disks) and fileserver, and some test hardware that I'm going to ignore. The most urgent things to replace are the HD based fileservers because our current disks are starting to die at an accelerating rate and you can't really get SATA drives with 512b sectors any more.

Thus a full scale replacement of the HD side requires eleven units (assuming we use the same case for fileservers and backends) and at least 84 WD Se drives. Replacing the SSD-based fileserver requires three units but no new data drives; our current SSDs are new enough to last us for a while. Due to the ZFS 4K sector mess we have to replace hardware in units of 'one fileserver and its backends', ie three units and 24 HDs. I'd like three units of test hardware (a fileserver and two backends), but I suspect we can't afford that.

(The current SSD-based fileserver has three backends for reasons that boil down to hardware issues with our current SSD enclosures. We wouldn't need to replicate this with new hardware.)

I'm going to skip doing a tentative costing out of all of this for fuzzy reasons. Interested parties can use the item and quantity counts here to do it for themselves.

Disclaimer: We haven't gone through any sort of competitive evaluation process to select this particular set of hardware out of the vast universe of possible hardware that meets our general specifications. We've just found hardware that meets our needs, has prices that seem sane, and that works in our testing (so far). As such I can't say anything about whether or not this would be your best and/or cheapest option in this area. We've also deliberately chosen not to put too many disks in one single physical unit (or to use disks that are too large, partly because of a desire to keep up our IOPs).

Sidebar: software and other details

We'll use some Linux with our usual iSCSI target software on the backends. The frontends will run OmniOS (and use ZFS). Using a single CPU core on the fileservers may strike some people as eye-raising, but we aren't going to be touching ZFS dedup at all and after thinking about some of the issues involved I don't think we want compression either. This makes me feel that dual-core would be overkill.

(I've tested both Linux and OmniOS on this hardware and they work, although tuning 10G performance is clearly going to be interesting.)