How we set up our iSCSI target servers

October 7, 2008

The SAN backend for our new fileserver environment is Linux servers acting as iSCSI targets. There's two sides to their setup; the hardware and the software.

Hardware-wise, the biggest problem was that there are essentially no generic servers (especially inexpensive ones) that have enough disks. So we punted; we're using 'generic' 1U servers, specifically SunFire X2100 M2s, with the iSCSI data disks in external ESATA-based enclosures. Our current enclosures give us twelve (data) disks in about 5U (counting both the enclosure and the server) at a reasonable cost.

(I would give a brand name but our current supplier recently told us they'd discontinued the model we had bought, so we're evaluating new enclosures from various people.)

Each server is connected to a single enclosure with a PCI-E card, currently SiI 3124-based ones. This requires a kernel that supports (E)SATA port multipliers, which only really appeared in 2.6.24 (for SII based cards) or 2.6.26 (for Marvell-based ones), and means that we have to build our own custom kernels. Software-wise, the servers run Red Hat Enterprise Linux 5 with the custom kernel and IET added as the iSCSI target mode driver, and are configured with mirrored system disks just in case.

(Recent versions of RHEL 5 have some support for SII based (E)SATA port multipliers, but 2.6.25 is what we've tested with and its support seems to work better.)

We don't do anything fancy with the iSCSI data disks (no software RAID, no LVM). Each disk is partitioned into even chunks of approximately 250 GBytes per chunk and then exported via iSCSI; we make each disk a separate target, and then each partition a LUN on that target. This makes life simpler when managing space allocation and diagnosing problems. (It also makes the IET configuration file contain a lot of canned text.)

We've decided to handle iSCSI as if it was a SAN, so we do not run iSCSI traffic over our regular switch fabric and VLANs. Instead, all of the iSCSI servers are connected to two separate iSCSI 'networks', which in this case means a dedicated physical switch and an unrouted private subnet; this gives us redundancy in case of switch, cable, or network port failure, and some extra bandwidth. Each server also has a regular 'management' interface on one of our normal internal subnets, so that we can ssh in to them and so on.

(Since they are X2100s, they also have a dedicated remote management processor on yet another internal subnet.)


Comments on this page:

From 68.236.185.94 at 2008-10-07 12:38:24:

That's really neat. Interesting!

What data do you put on them? I assume, since it's not RAID, that it's backup/archival/etc?

I've been looking for a way to store email on the cheap, and that would do it.

I'd also be interested in reading a post about which drive enclosures you end up using. Thanks!

Matt Simmons
Standalone Sysadmin

By cks at 2008-10-08 01:10:42:

The short answer is that we do the RAID in the SAN frontend machines for various reasons; the SAN frontends are Solaris ZFS-based NFS fileservers.

Written on 07 October 2008.
« A problem with Python's help()
How we set up our Solaris ZFS-based NFS fileservers »

Page tools: View Source, View Normal.
Search:
Login: Password:

Last modified: Tue Oct 7 01:33:46 2008
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.