Gotchas with IET that I have encountered

As a followup to yesterday's entry, I admit that I have run into a few issues with IET, the iSCSI target software that we wound up with. In a spirit of full disclosure, here they are:

• as I've mentioned before, IET wants a decent amount of memory; our current machines have two to two and a half gigabytes of RAM, which seems to be enough to survive even rather extreme stress testing. Less was definitely not enough.

• I experienced lockups when Solaris 10 x86's Solaris Volume Manager was talking to IET targets configured in 'blockio' mode (the same physical targets configured in 'fileio' worked fine, but performed worse). Our ultimate resolution for this was to use ZFS instead.

• IET only has rudimentary tools (some plain text files in /proc) for reporting its current configuration and state.

• although it can serve raw disks (and raw partitions), IET is not hotplug aware and so does not entirely recognize things like a disk being removed and reinserted; you can wind up with a situation where IET lists a target as present but can't actually talk to it. The situation is fixable but may be hard to find, because IET doesn't always report errors and there is no way to see if a given target is really working short of getting an initiator to try to talk to it.

• the IET daemon has no support for reloading its configuration file, and you cannot safely stop and restart it. This is not a fatal problem, because there is a program that can make most changes to the live configuration. However, the program doesn't update the configuration file, so it is possible to wind up with a running configuration that is completely different from what you'll get if the system reboots.

Okay, that's a simplification. The problem with stopping and starting the IET daemon is that this stops and restarts iSCSI service. If your initiators can talk to the server machine during this stop and restart, they are probably going to try to reconnect fast enough that they get a 'connection refused' error, and at least the Solaris 10 initiator reacts badly to this. (It gives up on the iSCSI target, presumably on the rational basis that, well, the target told it to go away.)

(There is also a smaller issue, in that while it was shutting down the daemon at least used to give strange and alarming replies to clients that reconnected sufficiently fast. I believe that has been fixed in the latest version, but I'm not sure.)

The easy way around this is to use iptables to block access to the iSCSI target (by blocking access to TCP port 3260) while the daemon stops and restarts. Solaris 10's initiator seems to deal with a stall and reconnect sequence reasonably well, possibly because this mimics what you see if the target reboots on you.

Some people will consider IET's basic plain text configuration file to be a drawback, but I don't.

Technically it is a drawback that IET doesn't support something called 'multiple connections per session' (MC/S) that is useful but not essential for load balancing and failover. However, the Solaris 10 initiator doesn't support MC/S either, so I'm not actually missing anything. (My impression is that MC/S is not very widely implemented in general, either in targets or initiators.)