(Technically it may survive with exactly half of the metadb replicas; I can't test right now.)

It survives (i.e., doesn't panic) with exactly half, but operator intervention is needed to bring the system back up if it goes down: the system boots single-user so that you can fix the problem.

Clearly, if your system had only two disks (or two sets of disks on redundant controllers) with all data mirrored between them, you wouldn't want the system to panic if one of them went down.

(There is an unsupported, undocumented /etc/system tunable to allow the system to boot with exactly 50% of its metadbs, but it's intended specifically for to allow HA systems with mirrored root disks to still come up if one of its disks fails; it's not intended to be used with anything except the root disk.)

The net result is that if you are using Solaris Volume Manager to manage iSCSI-based storage in metasets, you need to build metasets that include disks (logical or otherwise) from at least three different iSCSI targets or the loss of connectivity to a single target will kill your entire machine.

I think two should be sufficient, although three will improve the odds of the machine coming back up unattended if a panic or reboot does occur.

(And you need to carefully balance the number of metadb replicas across all of your targets so that one target doesn't have too many replicas.)

And, ideally, balance them with respect to other failure modes as well, where practical. For example, if 2/3 of your disks are accessed through one network path, and the other 1/3 are available through a different set of switches, you may wish to try to balance the metadbs so that quorum is not lost if either network path goes down.

--Dan

One of the really tricky bits of balancing metadb replicas is that metaset seems to like putting one on each (logical) disk that you add to the metaset. Since you may add different numbers of logical disks from different targets, you'll need to remember to strip some of the metadb replicas off to maintain the balance between targets.

...you'll need to remember to strip some of the metadb replicas off to maintain the balance between targets.

Or, perhaps preferably, to put additional metadbs onto some drives to maintain the balance, rather than stripping them off. (This may require manually repartitioning those drives to make slice 7 larger so that it can hold the number of metadbs you want.)

Disksuite's policy of putting metadbs on every drive has a purpose: it assures that as long as even one drive in the metaset is accessible, the system can determine the membership of the metaset. (Otherwise, I don't believe there would be anything to prevent an administrator from accidentally adding the drive to a different metaset, causing confusion when the other drives in the original metaset come back up.)

--Dan

In our usage, all of the logical drives coming from a target are actually coming from the same underlying pool of RAID storage, so either all the logical drives are available or none of them are and having metadb replicas on multiple logical drives on the same target doesn't really get us anything.

(Ideally I could tell DiskSuite what was really going on and it would balance metadb replicas based on the underlying storage, but I'm not going to hold my breath for that.)