The hardware and basic setup for our third generation of ZFS fileservers
As I mentioned back in December, we are slowly working on the design and build out of our next (third) generation of ZFS NFS fileservers, to replace the current generation, which dates from 2014. Things have happened a little sooner than I was expecting us to manage, but the basic reason for that is we temporarily had some money. At this point we have actually bought all the hardware and more or less planned out the design of the new environment (assuming that nothing goes wrong on the software side), so today I'm going to run down the hardware and the basic setup.
After our quite positive experience with the hardware of our second generation fileservers, we have opted to go with more SuperMicro servers. Specifically we're using SuperMicro X11SPH-nCTF motherboards with Xeon Silver 4108 CPUs and 192 GB of RAM (our first test server has 128 GB for obscure reasons). This time around we're not using any addon cards, as the motherboard has just enough disk ports and some 10G-T Ethernet ports, which is all that we need.
(The X11SPH-nCTF has an odd mix of disk ports; 8x SAS on one PCI controller, 8x SATA on another PCIE controller, and an additional 2x SATA on a third. The two 8x setups use high-density connectors; the third 2x SATA has two individual ports.)
All of this goes in a 2U SuperMicro SC 213AC-R920LPB case, which gives us 16 hot swappable 2.5" front disk bays. This isn't quite enough disk bays for us, so we've augmented the case with what SuperMicro calls the CSE-M14TQC mobile rack; this goes in an otherwise empty space on the front and gives us an additional four 2.5" disk bays (only two of which we can wire up). We're using the 'mobile rack' disk bays for the system disks and the proper 16-bay 2.5" disk bays for data disks.
(Our old 3U SC 836BA-R920B cases have two extra 2.5" system disk bays on the back, so they didn't need the mobile rack hack.)
For disks, this time around we're going all SSD for the ZFS data disks, using 2 TB Crucial SSDs in a mix of MX300s and MX500s. We don't have any strong reason to go with Crucial SSDs other than that they're about the least expensive option that we trusted; we have a mix because we didn't buy all our SSDs at once and then Crucial replaced the MX300s with the MX500s. Each fileserver will be fully loaded with 16 data SSDs (and two system SSDs).
(We're not going to be using any sort of SAN, so moving up to 16 disks in a single fileserver is still giving us the smaller fileservers we want. Our current fileservers have access to 12 mirrored pairs of 2 TB disks; these third generation fileservers will have access to only 8 mirrored pairs.)
This time around I've lost track of how many of these servers we've bought. It's not as many as we have in our current generation of fileservers, though, because this time around we don't need three machines to provide those 12 mirrored pairs of disks (a fileserver and two iSCSI backends); instead we can provide them with one and a half machines.
Sidebar: On the KVM over IP on the X11SPH-nCTF
The current IPMI firmware that we have still has a Java based KVM over IP, but at least this generation works with the open source IcedTea Java I have on Fedora 27 (the past generation didn't). I've heard rumours that SuperMicro may have a HTML5 KVM over IP either in an updated firmware for these motherboards or in more recent motherboards, but so far I haven't found any solid evidence of that. It sure would be nice, though. Java is kind of long in the tooth here.
(Maybe there is a magic setting somewhere, or maybe the IPMI's little web server doesn't think my browser is HTML5 capable enough.)