I mentioned recently that we've written our own system to handle ZFS spares. Before I describe it, I wanted to write up something about why we decided to go to the extreme measure of discarding all of ZFS's own spare handling and rolling our own.

First off, note that our environment is unusual. We have a lot of pools and a relatively complex SAN disk topology with at least three levels, as opposed to the more common environment of only a few pools and essentially undifferentiated disks. I expect that ZFS's current spares system works much better in the latter situation, especially if you don't have many spare disks.

Our issues with ZFS's current spare system include:

• it has outright bugs with shared spares, some of them fixed and others not (we had our selfish pool, for example).

• because of how ZFS handles spares, we've seen ZFS not activate spares in situations where we wanted them activated.

• ZFS has no concept of load limits on spares activations. This presents us with an unenviable tradeoff; either we artificially limit the number of spares we configure or we can have our systems crushed under the load of multiple simultaneous resilvers.

  (We've seen this happen.)

• ZFS doesn't know how we want to handle the situation where there are too few spares to replace all of the faulted disks; instead it will just deploy spares essentially randomly. (This also combines with the above issue, of course.)

• there's no way to tell ZFS about our multi-level disk topology, where there are definitely good and bad disks to replace a given faulted disk with.

Many of these are hard problems that involve local policy decisions, so I don't expect ZFS to solve them out of the box. Instead ZFS's current spares system deals with the common case; it just happens that the common case is not a good fit for our environment.

(I do fault ZFS for having no support for this sort of local additions. I don't necessarily expect a nice modular plugin system, but it would be nice if ZFS had official interfaces for extracting information in ways that are useful for third party programs. But that's really another entry.)