Chris's Wiki :: blog/tech/KernelISCSITargetBias

It turns out I'm biased towards kernel iSCSI target implementations

January 9, 2013

In the process of writing up why we wound up using Linux for our iSCSI targets I wound up looking up the state of iSCSI target implementations on FreeBSD. It looks like they have one but it also seems to be a purely user-mode implementation (as a daemon), and when I realized that I discovered that I have an odd visceral bias against user-mode iSCSI targets and towards an in-kernel implementation.

I find this odd, partly because I don't know where this bias comes from. iSCSI is a sufficiently complex protocol to be the kind of thing I'd normally like to have out of the kernel (in code that's easier to manage and deal with), I don't have any bad experiences with a user-mode iSCSI target implementation (I don't think I've ever used one), and in fact when I looked into ATA-over-Ethernet (which you might expect me to have the same reaction to) I actually considered writing my own user-level implementation for production use.

I don't think there's any strong technical reason to (strongly) favour an in-kernel iSCSI target implementation. Depending on kernel and user level APIs such an implementation might have an easier time of doing efficient disk and network access, but I don't think it's likely to have a huge effect (in part because there's lots of spare CPU with current kernel implementations). In theory an in-kernel implementation might do less copying around of disk buffers. But given the oddities of the protocol you may well not be able to achieve this with any particular iSCSI initiator, or even in general, simply because the protocol (plus TCP) already requires you to slice and dice things around.

Despite all of this, a user-level iSCSI target implementation just makes me reflexively nervous. For no good reason I trust kernel ones more. Maybe I'm not trusting a user-level implementation to be able to manage enough concurrency to drive a big system at full capacity.

(This is one of those entries where I don't have any answers, just a realization that I want to note down. If I get the chance, I now want to explore the performance of a good user-level iSCSI target implementation.)

Comments on this page:

From 90.155.35.116 at 2013-01-09 19:29:55:

Hrm, interesting question. I know that I don't like userspace NFS servers, because last time I tinkered with one it went horribly horribly wrong. I suspect I think of NFS and iSCSI as being kinda sorta the same. Thus I'd be tempted to believe that userspace iSCSI is horrible because userspace NFS was horrible (once upon a time).

-- Perry Lorier

By cks at 2013-01-11 14:17:44:

A somewhat belated note: userspace NFS servers are basically doomed for reasons I wrote up in NFSUserServerProblem. Generalizing from them to how well other userspace servers will work is thus probably going to be misleading (to put it one way).

From 86.135.246.217 at 2013-01-22 06:22:29:

maybe it is because user space iscsi target will be generally slower than an in-kernel implementation?

Also I recommend to look into COMSTAR in Solaris 11 or Illumos. This is in-kernel implementation which can use ZFS underneath.

By cks at 2013-01-23 17:57:25:

I don't think that there's any particular reason for a user space iSCSI target to be noticeably slower than a kernel one except under very unusual circumstances. Our iSCSI backends (with kernel targets) can saturate two 1GB Ethernet links with quite low CPU usage (on what is now several years out of date hardware); while a user mode target might use more CPU, it clearly has a lot of room to do so without coming anywhere near CPU saturation. Any increased time taken by more kernel/user mode transitions is likely to vanish into the noise of HD latency or even SSD latency.

For these things to make a difference I think you would have to be running a kernel mode target very near the limits of hardware performance and I'm honestly not sure how close you can come to those (and how you'd do it). You need to be pushing so much data or using something that responds so fast (or both) that what are likely to be relatively modest amounts of extra CPU usage, context switches, and memory copies push you over a performance edge.

Written on 09 January 2013.

«	Why we wound up using Linux for our iSCSI targets

The fundamental problem faced by user-level NFS servers