Wandering Thoughts: Recent Entries For 2011/12/10

ZFS pool activation and iSCSI (part II)

Here's an interesting question that should have occurred to me much earlier:

Why is a bug about boot-time ordering between iSCSI disk discovery and ZFS pool activation fixed with a kernel patch?

I don't have a sure answer to this; the best I can do is a theory. But before we get there, let's talk about how ZFS pool activation seems to connect up with the Solaris iSCSI initiator.

At the SMF level, the iSCSI initiator is svc:/network/iscsi/initiator. Nothing explicitly depends on it (at least according to 'svcs -D'). Despite this, on our S10U8 machines it finishes starting immediately before svc:/system/filesystem/local does (which is exactly what you want, since the latter SMF service seems to be what starts ZFS pools). Exactly why SMF uses or enforces this order is opaque to me. For that matter it's not clear if SMF itself is enforcing the order; because SMF only shows the order by end time, not by start time, it's possible that the start order is different than the finish order.

(A great deal of SMF is opaque, annoying, or both.)

Now it's time for theorizing.

If we take SMF at its word, there is no explicit ordering dependency in SMF. Any ordering we get is either a lucky coincidence or enforced by something else, and I don't believe it's a lucky coincidence. The obvious candidate to enforce an ordering is the kernel, since it handles both the iSCSI and ZFS parts of all of this. It would make a kind of sense if the kernel delayed ZFS pool activation until iSCSI discovery had finished; it's very analogous to how kernels often delay things for SCSI disk discovery. Given that iSCSI disk discovery can be quite protracted, it would also make sense if at some point a clever Sun kernel developer broke that absolute dependency so that the boot could still proceed even if iSCSI discovery was taking ages; such a dependency break would match the symptoms we saw here, where 'zfs mount -a' ran after iSCSI discovery had started but before it had finished. The fix for this kernel dependency issue would of course be another kernel change.

(Since Oracle no longer updates the OpenSolaris source code it's impossible to verify this theory. Besides, my patience for spelunking Solaris kernel code is pretty close to being exhausted.)