Wandering Thoughts archives

The problem with initial ramdisks

Here's one of my Linux peculiarities: I don't like initrds. Not because they're inefficient or anything like that, but because they are another moving part in an already complicated boot process. Worse, they are generally a black box moving part; it is pretty difficult to peer inside an initrd to see what it is doing and to make sure that it is correct.

In fact, initrds are so hard to inspect and manipulate that almost no one does, not even distributions. Instead initrds are 'manipulated' by recreating them from scratch, and aren't generally inspected at all; if you have doubts about an initrd, you remake it. And building them is magic, so you have to trust that the magic was done right (and the magic is supplied by your distribution, as it's beyond most people's abilities to hand-build an initrd environment).

(Partly this is because initrds are so general and no reliable standards have emerged for how to organize them. This means that just looking at a file and directory listing is no real help; in order to know what's going on, you have to crack a particular initrd apart and read the scripts involved.)

But let's set aside my issues with the common implementations, because the big problem is that moving parts are inherently fragile, especially when they must exactly match up with other moving parts. An initrd is another point of failure; if its contents aren't exactly right, my system won't boot. (And there are a fair number of ways for the contents to wind up not exactly right.)

This might be worth it if I got something for this extra fragility, but I don't, at least so far. So I am left feeling a little bit grumpy about the whole issue, especially when distributions require initrds.

Gotchas with IET that I have encountered

As a followup to yesterday's entry, I admit that I have run into a few issues with IET, the iSCSI target software that we wound up with. In a spirit of full disclosure, here they are:

• as I've mentioned before, IET wants a decent amount of memory; our current machines have two to two and a half gigabytes of RAM, which seems to be enough to survive even rather extreme stress testing. Less was definitely not enough.

• I experienced lockups when Solaris 10 x86's Solaris Volume Manager was talking to IET targets configured in 'blockio' mode (the same physical targets configured in 'fileio' worked fine, but performed worse). Our ultimate resolution for this was to use ZFS instead.

• IET only has rudimentary tools (some plain text files in /proc) for reporting its current configuration and state.

• although it can serve raw disks (and raw partitions), IET is not hotplug aware and so does not entirely recognize things like a disk being removed and reinserted; you can wind up with a situation where IET lists a target as present but can't actually talk to it. The situation is fixable but may be hard to find, because IET doesn't always report errors and there is no way to see if a given target is really working short of getting an initiator to try to talk to it.

• the IET daemon has no support for reloading its configuration file, and you cannot safely stop and restart it. This is not a fatal problem, because there is a program that can make most changes to the live configuration. However, the program doesn't update the configuration file, so it is possible to wind up with a running configuration that is completely different from what you'll get if the system reboots.

Okay, that's a simplification. The problem with stopping and starting the IET daemon is that this stops and restarts iSCSI service. If your initiators can talk to the server machine during this stop and restart, they are probably going to try to reconnect fast enough that they get a 'connection refused' error, and at least the Solaris 10 initiator reacts badly to this. (It gives up on the iSCSI target, presumably on the rational basis that, well, the target told it to go away.)

(There is also a smaller issue, in that while it was shutting down the daemon at least used to give strange and alarming replies to clients that reconnected sufficiently fast. I believe that has been fixed in the latest version, but I'm not sure.)

The easy way around this is to use iptables to block access to the iSCSI target (by blocking access to TCP port 3260) while the daemon stops and restarts. Solaris 10's initiator seems to deal with a stall and reconnect sequence reasonably well, possibly because this mimics what you see if the target reboots on you.

Some people will consider IET's basic plain text configuration file to be a drawback, but I don't.

Technically it is a drawback that IET doesn't support something called 'multiple connections per session' (MC/S) that is useful but not essential for load balancing and failover. However, the Solaris 10 initiator doesn't support MC/S either, so I'm not actually missing anything. (My impression is that MC/S is not very widely implemented in general, either in targets or initiators.)

My experiences so far with Linux iSCSI target software

In iSCSI terminology, initiators (clients) talk to disks on targets (servers). Thus, iSCSI target software is software that lets your machine serve disks over iSCSI to other machines, while iSCSI initiator software lets your machine access iSCSI disks. Linux has essentially had a standard iSCSI initiator for some time (including the necessary kernel level driver integrated into the mainline kernel), but there is no standard iSCSI target software; instead there are various different projects, such as IET, SCST, and STGT.

(This situation is confusing, partly because STGT appears to do iSCSI entirely in user-mode code, while IET and SCST do at least part of the work in kernel drivers. For my purposes this distinction didn't matter, since RHEL 5 didn't have any of them packaged and integrated.)

I wound up picking IET, partly because it was what came up first in web searching and partly because it was relatively clear how to get it configured and serving space via iSCSI (which is more than I could say for the other two). It does have the downside of having its own kernel module, but I didn't have problems getting it compiled and working with the kernel we wanted to use.

(We are unlikely to change the kernel without a lot of work; our iSCSI target servers are not user accessible and are basically closed black boxes, so most reasons to update the kernel don't apply.)

IET does fine on the performance front, in that IET does not seem to be a limiting factor. It is easy to get wire bandwidth with CPU capacity to spare, even without jumbo frames, and I have seen a single server saturate two gigabit links. (Jumbo frames do reduce CPU consumption, but our iSCSI target servers don't have anything else to do with those CPU cycles anyways.)

I experimented briefly with SCST but failed to figure out how to make it actually serve disks over iSCSI in any relatively simple way. I took this as a sign that it was too complex (and too under-documented) to make me happy and did not investigate further.

(Note that there are various political disagreements between the iSCSI target software projects that make it very challenging to find unbiased comparisons. I'm not going to claim that this is one, partly because I have done no sort of competitive evaluation; I was only interested in finding an iSCSI target driver that worked well enough, so once I established that IET did, I stopped.)