What I currently know about Fibrechannel versus iSCSI versus AoE

February 19, 2007

We're planning a significant capacity increase for our local SAN storage pool, which means that we've been trying to figure out which SAN technology we want to go with. We don't have high performance IO needs, so we're going for bulk storage: large SATA disks in RAID 5 in some sort of SAN RAID controller. We plan to use Solaris 10 on x86 for our NFS servers that will use the SAN, with DiskSuite for failover. DiskSuite has to own full disks if it's going to do failover, not just partitions, so we need our SAN RAID controllers to export logical LUNs.

(The local opinion is that we trust Solaris more than Linux as a NFS server, plus it has DiskSuite.)

There are three possible choices: Fibrechannel, iSCSI, and ATA-over-Ethernet. Of these:

  • iSCSI and AoE can be had for about $5k for a 15-bay 3U RAID controller, in the form of the Promise VTrak M500i or the Coraid SR1521. You buy commodity SATA disks yourself from your cheap source of choice.
    (There are probably other vendors for 15-bay 3U iSCSI controllers; we haven't looked very hard.)
  • FC costs about $5.5k and up for a 12-bay 2U RAID controller. You have to buy the disks through the controller vendor, at a not insignificant markup.
  • as a result, FC costs about twice as much per terabyte as iSCSI or AoE.
  • people on campus have positive experience with all three options, and none of them have blown up yet.

  • AoE and iSCSI cost much less than Fibrechannel to add stuff to later, because Fibrechannel switches are really expensive and thus a) you tend not to have many spare switch ports and b) getting more switch ports is expensive.
  • similarly, it is much cheaper to have redundancy and spares for your AoE or iSCSI switching fabric.
  • for at least AoE, you want switches and machines that can do jumbo frames. This probably won't hurt for iSCSI either.

  • there is only one vendor of AoE RAID controllers, Coraid. Coraid's stuff currently does not do logical LUNs within a single disk array.
  • while Promise's stuff does do logical LUNs, it has some limits on how many you get within a single disk array. Fortunately we seem unlikely to run into them.
  • Coraid's management and monitoring software seems to be less advanced than Promise's, which will do things like mail you problem reports.

  • AoE has a much simpler specification than iSCSI, but this is somewhat misleading because the AoE spec doesn't say what ATA commands you must support in order to talk to common AoE implementations, and thus doesn't include a spec for them; in practice the AoE spec has to be considered to include some of the ATA spec itself.

  • Linux has both AoE and iSCSI drivers in the standard kernel.
  • Solaris 10 has standard iSCSI drivers, but no standard AoE ones. Coraid is sponsoring the development of an open source AoE driver, but it's currently only tested on SPARC systems, not x86 systems, and may not yet fully support ZFS (apparently ZFS needs the disk driver to support some new operations). However, it supports Solaris 7, 8, and 9 in addition to Solaris 10.
  • the Linux AoE driver is about 2,000 lines; the iSCSI driver is about 4,000 5,800 lines between the actual driver and the iSCSI library. The AoE driver is a straight block driver, the iSCSI driver is a SCSI driver.

  • there is a non-integrated AoE target driver for Linux called vblade. No one else has AoE target drivers. (Target drivers allow you to use a machine as a SAN RAID controller that exports storage to other machines.)
  • there is an integrated iSCSI target driver for Solaris 10, although it is not yet in official releases.
  • there are a number of Linux iSCSI target drivers; none are integrated (yet).

  • in general, iSCSI is more mature and widely supported than AoE.
  • Linux seems to have better support for AoE than for iSCSI, which is probably because AoE is simpler and has less peculiar bits. (There is a certain enterprisey smell about iSCSI.)

Since we are not interested in building our own SAN RAID controllers, we are almost certainly going to wind up with iSCSI; AoE is unsuitable on several grounds, and Fibrechannel costs too much for what we get. If we were building our own SAN RAID controllers out of PCs running in target mode, I would be very tempted by AoE because of the simplicity of all of the bits involved (and building our own would overcome several of the things that make AoE unsuitable).

Comments on this page:

From at 2007-02-21 02:40:48:

How confident are you that your SCSI drivers are hardened against direct network attack? The neat thing about iSCSI, from a defense perspective, is that it exposes SCSI command channels to network traffic.

It used to be if you wanted to crack a box using SCSI driver holes, you already had to be root to get access, so what was the point. iSCSI offers new and interesting ways into your kernel from outside your box. Limit who can talk to you accordingly.

By cks at 2007-02-21 08:47:19:

We're only looking at SAN solutions, so we'd only be running iSCSI on a completely isolated network (with a dedicated switch; we're not even going to try a VLAN on our existing switch fabric). We may even have to keep ordinary users off the fileservers entirely, so they can't start connecting to ports on the iSCSI controllers. iSCSI does have some degree of connection authentication, but I'm not sure I trust it that much.

(This is one attraction of AoE; since it isn't IP-based, you'd have to be root on the fileservers to start with in order to send packets to the AoE controllers.)

From at 2007-03-02 22:09:33:

You mentioned in a previous article that you were looking for a fast way to failover NFS services. If you are going to implement Solaris 10, you might consider deploying Sun cluster along with it. Sun cluster is free to use, supports ZFS (they support SVM and VxVM as well), and has an agent to start, stop, monitor and failover NFS services between cluster members. I support one NFS cluster running Solaris 10 11/06 and SC 3.2, and I coulnd't be happier with it.

- Ryan

From at 2009-01-06 10:17:14:

- the current price of aoe storage is $5k/24 slots

- coraid sr appliances do syslog for problem reports.

- only a few ata commands need to be implemented.

vblade only implements 5 commands (identify, read, read extended, write and write extended). aoe is very simple to program, configure or use.

- the solaris aoe driver supports x86.

- there are several vblade-style implementations, including one for the linux kernel. i wrote a cleanroom version of vblade for plan 9 in an evening.

- erik quanstrom

Written on 19 February 2007.
« Another aphorism of system administration
Getting around LiveJournal's new minimum page width »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Feb 19 22:33:40 2007
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.