Multi-mount protection and SAN failover

August 11, 2013

Suppose that you have some machines, some shared disks, and a filesystem that has what gets called multi-mount protection, where it tries to prevent being mounted on two different machines at the same time. Do you have enough to do reliable SAN failover in the face of a crashed machine? Unfortunately the answer is no.

The first problem is that common implementations of multi-mount protection are not necessarily fully reliable. While it is possible to do reliable locking with only read and write operations to a shared disk (see eg Dekker's or Peterson's algorithm), many MMP implementations do not go to this extent; instead they rely on statistical properties, such as that a check block will get written every so often by the server that owns the filesystem. This usually works but cannot be absolutely guaranteed to do so in the face of a machine that is in an unknown but broken state.

(At the simplest level the check block might be written by a separate 'check block writer' process that has gotten stuck somehow.)

The larger problem is that multimount protection is essentially solving the wrong problem for failover. For forced failover, system A legitimately owned the filesystem when it was healthy and active but is now in some unknown but unhealthy state. Since it is in an unknown state, it has not properly released its ownership and you cannot count on it to be inactive. You want to forcefully take ownership of the filesystem away from system A and make it so that system A will not write anything more to the filesystem, and you must do this without system A's cooperation (because it may not cooperate, since it is in a bad state).

At best multi-mount protection will tell you that system A does not seem to have recently written anything that MMP checks. It cannot assure you that system A will not do any writes in the future. To do this you must somehow forcefully fence system A away from filesystem writes, either with storage-level features or simply by a remote power off of system A.

You can get away with active, check-based MMP for SAN failover only if you trust things to not go too badly wrong, so that the check saying that system A is inactive is sufficient to guarantee that it actually is and that it will stay that way.

Sidebar: two levels of multi-mount protection

The basic level of multi-mount protection is simply an 'is active' marker of some sort in the filesystem; a system sets the marker when it mounts the filesystem and unsets it when the filesystem is unmounted. This sort of MMP doesn't help you at all in failover because system A is unlikely to have actually released the filesystem before it stopped working right.

The advanced level of MMP is something in the filesystem that is actively updated on a frequent basis. If you passively watch the filesystem and there are no updates to the marker over a sufficiently long time, you can conclude that the theoretical owner either doesn't exist any more or at least is not working right.

Written on 11 August 2013.
« The importance of names, illustrated through my mistake
The feature (or features) I really want added to xterm »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun Aug 11 00:01:52 2013
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.