An interesting mistake with ZFS and iSCSI
First I'll show you the symptoms, then I'll explain what I did to shoot myself in the foot:
# zpool create tank01 c0t38d0 # zpool create tank02 c0t39d0 # zpool replace tank01 c0t38d0 c0t42d0
(Time passes, the resilver finishes, and
zpool status tank01 shows no
use of c0t38d0.)
# zpool attach -f tank02 c0t39d0 c0t38d0
invalid vdev specification
the following errors must be manually repaired:
/dev/dsk/c0t42d0s0 is part of active ZFS pool tank01. Please see zpool(1M).
All of these disks are iSCSI disks being exported from a Linux machine.
The error condition was persistent, lasting through reboots,
zpool import and so on, while at the same time nothing
said that c0t38d0 was in use or active.
How I shot myself in the foot is simple: I configured all of the iSCSI
disks with the same
When I set up the Linux target software, I'd assumed that its 'SCSI ID'
was something like a model name, partly because there's also a
parameter for each disk's nominal serial number. I was totally wrong; it
needs to be a unique identifier, just like the
ScsiSN values (and if
left alone, the target software would have handled it).
What is presumably going on is that ZFS noticed that c0t38d0 has the
ScsiId as c0t42d0, concluded that they were two names for the
same actual disk (which is easily possible in a multi-path setup), and
sensibly refused to let me shoot myself in the foot. The one thing I
don't understand is why it only happens with c0t42d0, which is the last
of the iSCSI disks.