Wandering Thoughts archives

Some notes about iSCSI multipathing in Solaris

Our new fileservers have multiple network paths to the backend iSCSI storage, which means that we needed to set up some sort of multipathing. Theoretically, there's at least three different ways of doing this: bonding multiple network interfaces together, a native iSCSI feature for this called 'multiple connections per session' (MC/S), and high level, cross device multipathing. In practice, the complexity of getting network bonding working in a cross-vendor environment gave me hives and neither Solaris nor our Linux iSCSI targets support MC/S, so we've using MPxIO, Solaris's high level multipathing system.

(My impression is that MC/S would be the best solution if both ends supported it for reasons that don't fit into this entry.)

MPxIO is a little peculiar, especially in its interactions with iSCSI; it seems to have really been built for a previous generation of technology and drivers, and not really adopted well to iSCSI. It works, but various things are a little bit awkward. The bits that spring particularly to mind are:

• persuading MPxIO to actually multipath iSCSI things is a pain, as it requires you to stuff magic options into /kernel/drv/scsi_vhci.conf. Fortunately our target software always advertises a constant vendor and product ID.
• the mpathadm command is basically useless.

• the combination of MPxIO and iSCSI leaves you with no way to explicitly control which network path is the active path. This makes it a good thing that MPxIO defaults to round-robin load balancing.

• you have to remember to explicitly set the Solaris iSCSI initiator to make more than one connection per target.

• under some circumstances, the Solaris iSCSI initiator will make two connections and multipathing will look perfectly happy, except that the two connections are over the same network path.

The last issue seems to happen if only one of the two networks is available when the iSCSI code is bringing up the target. One way to have this happen is that if you add statically configured iSCSI targets (ie you have targetA,IP1 and targetA,IP2), the first target IP address added will get both connections and the second target IP address will see none. You can cure the situation by temporarily changing the initiator to only make one connection per target and then setting it back to the normal value, but this temporarily drops the second connection to all targets.

(The only way I've found to notice this is to pay close attention to the IP addresses shown in 'iscsiadm list target -v' and spot the duplication.)