An important gotcha with iSCSI multipathing in Solaris 10

March 16, 2009

Here's something important to know about Solaris's MPxIO multipathing: MPxIO identifies disks only by their serial numbers and identifiers. So if two Solaris devices have the same serial number, MPxIO concludes that they are two paths to the same physical disk; it has no actual knowledge of underlying path issues, such as iSCSI target identifiers.

This matters a great deal on iSCSI, because at least some iSCSI initiators have serial numbers that are set in software. If you accidentally duplicate some serial numbers between different disks, Solaris's MPxIO will happily decide that they are all the same disk and start distributing IO among them. The result will not make your filesystem very happy. (If you are using ZFS, you have probably just lost the entire pool and possibly the system as well.)

(This is similar to my previous mistake along these lines, but much bigger. I am fortunate that I made this mistake in testing.)

Or in short: when you set up iSCSI targets, make very sure that they have unique SCSI serial numbers et al.

It's hard to fault MPxIO for this behavior, since part of MPxIO's job as a high level multipathing system is to join together the same drive when it's visible over multiple different transport mediums (for example, a drive that is visible over both FibreChannel and iSCSI, however peculiar that may be). Still, it makes adding new targets a bit nerve-wracking, since I know that one mistake or oversight with the configuration of a new iSCSI backend may destroy a pool on an unrelated set of storage.

(This is where I wish Solaris (and our iSCSI backends) had iSCSI specific multipathing, which would avoid this problem because it knows that two completely different targets can never be the same disk.)


Comments on this page:

From 150.142.232.7 at 2009-06-02 14:28:53:

Hi Chris, I know this blog entry is kind of old, but you mention "when you set up iSCSI targets, make very sure that they have unique SCSI serial numbers". Is there a zfs or iscitadm command to do this? I've been pulling my hair out trying to use zfs/iscsi as a target for VMware ESX.

Any info would be greatly appreciated.

thanks,

paulrlyons at gmail dot com

By cks at 2009-06-05 12:19:42:

I don't know of any explicit command to set the SCSI serial number information for Solaris targets, so I assume that ZFS and iscitadm do this automatically (making up some sort of GUID for this info).

The last time I looked the Solaris iSCSI target stuff was not what I could call mature, but that was a while ago and things may have gotten better since S10U4.

Written on 16 March 2009.
« Complex data structures and the two sorts of languages
Principles of email in the modern age »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Mar 16 23:47:29 2009
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.