Wandering Thoughts archives

Our problem with iSCSI connections at boot on OmniOS

You might perhaps wonder why I recently needed to run a script when our OmniOS machines booted. As it happens, we sometimes have a little problem with our iSCSI networking when we reboot a system, and we would like to know about it right away. First, the high speed summary of iSCSI on our ZFS fileservers is that fileservers connect to their iSCSI backends over two separate and thus redundant networks. At a mechanical level this is done by statically configuring each iSCSI target disk twice, one over each network, joining them together with standard OmniOS multipathing (set to round-robin), and then telling the OmniOS iSCSI initiator that it should make two connections to each target with 'iscsiadm modify initiator-node -c 2' (here's a longer writeup).

What we want and expect is that those two connections to each target should be made over different networks. And most of the time this works. However, some of the time a system will boot up with all of its connections to some or even all of the targets going over only a single network. Usually there will still be two connections but both will be over the same network, which costs us both redundancy and bandwidth.

(It's possible that OmniOS would make a new connection over the other network if the first one died, but this isn't something we exactly want to bet on.)

Because nothing actually breaks when the system is like this (at least when both iSCSI networks are working), it's possible for fileservers to quietly stay in this state for some time. Once we got disturbed enough by this fact, we wrote a script on the backends that checks for this, but only once a day. We decided that we'd like to know faster than that for the most common case, where this unbalanced iSCSI usage happens at boot time and can be detected right after boot. That led to needing a boot time service to run the script and wound up with me deep in SMF for the first and hopefully last time.

By the way, this is not directly OmniOS's fault; it's something that's been happening in Solaris for some time. My assumption is that this problem has at least something to do with the tangled way that Solaris has always brought up iSCSI disks at boot time, such that the OmniOS iSCSI initiator is attempting to bring up the two connections we told it to make at a time when only one network is available.

(Perhaps I should file this as an OmniOS and/or Illumos bug, but somehow I doubt it would get much attention.)

Sidebar: How we fix this

In an ideal world, you could fix this simply by telling OmniOS to switch to having only one connection per target, then go back to two connections per target; OmniOS would notice that it had two networks available and that it would be smart to make that second connection over the other network. Sometimes this even works. Often it doesn't, though.

When it fails to work, what has worked for us is to remove entirely the static target configuration for the network that is not being used, drop to one connection per target, re-add all of those removed static target configurations, and go back to two connections per target. Fortunately we have scripts that generate most of the necessary commands.

Some notes on SMF manifests (on OmniOS) and what goes in them

Recently, I needed to create a SMF manifest to run a script at boot. In most init systems, this is simple. SMF is not most init systems. SMF requires services (including scripts run at boot) to be defined in XML manifests. Being XML, they are verbose and picky, but fortunately there are some good general guidelines on what goes in them; the one I started from is Ben Rockwood's An SMF Manifest Cheatsheet. But there are a number of things it didn't say explicitly (or at all) that I had to find out the hard way, so here's some notes.

First, on OmniOS you'll find most existing SMF manifests under /lib/svc/manifest, especially /lib/svc/manifest/system. If you get confused or puzzled on how to do something, itt's very worth raiding these files for examples.

What both Ben Rockwood's writeup and the documentation neglects to mention is that there is a fixed order of elements in the SMF manifest. The manifest is not just an (XML) bag of properties; the elements need to come in a relatively specific order. You can get all sorts of puzzling and annoying errors from 'svccfg validate' if you don't know this.

(The error messages probably make total sense to people who understand XML DTD validation. I am not such a person.)

For just running a script, everyone seems to set things so there is only a single instance of your SMF service and it's auto-created:

<create_default_instance enabled='false' />
<single_instance/>

(This comes right after the opening <service> tag.)

There is probably an art to picking your SMF dependencies. I went for overkill; in order to get my script run right at the end of boot, I specified /system/filesystem/local, /milestone/multi-user, /milestone/network, and for local reasons /network/iscsi/initiator. 'svcs' defaults to listing services in start order, so you can use that to fish around for likely dependencies. Or you can look at what similar system SMF services use.

(It turns out that you can put multiple FMRIs in a single <dependency> tag, so my SMF manifest is more verbose than it needs to be. They need to have the same grouping, restart_on, and type, but this is probably not uncommon.)

Although you might think otherwise, even a single-shot script needs to have a 'stop' <exec_method> defined, even if it does nothing. The one that services seem to use is:

<exec_method type='method'
             name='stop'
             exec=':true'
             timeout_seconds='3' />

The timeout varies but I suspect it's not important. Omitting this will cause your SMF manifest to fail validation.

If you just want to run a script from your SMF service, you need what is called a 'transient' service. How you specify that your service is a transient one is rather obscure, because it is not something you set in the overall service description or in the 'start' exec_method (where you might expect it to live). Instead it's done this way:

<property_group name='startd' type='framework'>
    <propval name='duration' type='astring' value='transient' />
</property_group>

This is directions for svc.startd, which is responsible for starting and restarting SMF services. You can thus find some documentation for it in the svc.startd manpage, if you already understand enough about SMF XML manifests to know how to write properties.

(Since it is an add-on property, not a fundamental SMF XML attribute, it is not to be found anywhere in the SMF DTD. Isn't it nice that the SMF documentation points you to the SMF DTD for these things? No, not particularly.)

Some documentation will suggest giving your SMF service a name in the /site/ overall namespace. I suggest using an organizational name of some sort instead, because that way you know that a particular service came from you and was not dropped in from who knows where (and it's likely to stand out more in eg 'svcs' output). Other people creating SMF packages are already doing this; for instance, pkgsrc uses /pkgsrc/ names.

(This is the kind of entry that I write because I don't want to have to re-research this. SMF was annoying enough the first time around.)

Sidebar: A quick command cheatsheet

SMF manifests are validated with 'svccfg validate <file>.xml'. Expect to use this often.

Once ready to be used, manifests must be imported into SMF, which is done with 'svccfg import <file>.xml'. If you specified that your service should default to disabled when installed (as I did here), you then need to enable it with the usual 'svcadm enable /you/whatever'.

In theory you can re-import manifests to pick up changes. In practice I have no idea what sort of things are picked up; for example, if you delete a <dependency> block, does it go away in the imported version when reimported? I'd have to experiment (or know more about SMF than I currently do).

Your imported SMF manifest can be completely removed with 'svccfg delete /you/whatever'. Normally you'll want to have disabled the service beforehand. The svccfg manpage makes me slightly nervous about this in some circumstances that are probably not going to apply to many people.

(Svccfg has an export operation, but it just dumps out information, it doesn't remove things.)