We're (still) looking for a new SAN, so I have recently been doing some work with a homebrew iSCSI testing environment built from ordinary servers. The experience leaves me with a simple question: why does anyone buy special iSCSI accelerator network cards?

(These are sometimes called 'iSCSI HBA cards'.)

I ask because my ordinary test environment, with commodity server NICs and no tuning besides 9,000 byte jumbo frames, can basically saturate gigabit Ethernet doing bulk reads over iSCSI. Through the filesystem, no less. Neither end is hurting for CPU; the iniator machine shows 75% idle, the target machine show 40% idle.

(I don't care about the target machine's CPU all that much, since it has no other role in life besides being an iSCSI target; what really matters is iniator CPU usage.)

To me this smells an awful lot like the hardware RAID situation, where you have to be both CPU constrained and doing a large volume of IO in order for an iSCSI HBA to make any sense. My suspicion is that this is not very common, and thus that a lot of people are being scared into buying iSCSI accelerators that they don't need.

(The one possible exception I can think of is if you are using iSCSI to supply disk space to virtualized servers. Still, I can't help but think that such a machine is maybe a bit overcommitted.)

Sidebar: my test environment

If for no other use than my future reference:

• the initiator machine is a SunFire X2100 M2 running Solaris 10 U4 with ZFS and the native Solaris iSCSI iniator.
• the target machine is an HP DL320s running Red Hat Enterprise 5 (in 64-bit mode) with the IET driver. The iSCSI-exported storage is LVM over software RAID 6 across 11 80GB SATA disks.

(The HP's native hardware RAID 6 turned out to have abysmal performance, so I fell back to Linux software RAID, which performs fine.)

The Solaris machine is using an Intel gigabit NIC (via a Sun expansion card); the HP machine is using one of the onboard Broadcom BCM5714 ports. Linux enabled RX and TX checksumming and scatter-gather support on the Broadcom, and I don't know what Solaris did. The machines are connected through a Netgear switch.

(There is a rant about switches that claim to support jumbo frames but don't that could go here.)