2010-04-05
An important thing about how ZFS handles spares
There's an important thing about how ZFS handles spares that's implicit in my first entry, but it's such an important thing that it deserves to be said explicitly:
ZFS only activates spares when faults happen.
In a traditional spares implementation, spare activation is driven by the state of the RAID array; if the array is unhealthy and there is a spare, the spare is activated. In ZFS, spares are activated (only) in response to fault events; if the right sort of pool fault happens and there is a spare, it is activated. This makes ZFS very different from traditional RAID arrays in that a ZFS pool can have both problems and available spares but not activate them because there is no fault event to cause it to do so.
(The short version of this difference is that traditional spare handling is state-driven, while ZFS's spare handling is event-driven.)
There are two situations where this matters a lot. First, if you add a spare to a pool with some problem, ZFS will not activate it because the fault event is now in the past. Effectively, you get exactly one chance to have a spare activated, when the original fault happens, and that's it; if anything goes wrong at the time, you will have to intervene by hand.
Second, there are a number of situations where Solaris will discover
pool problems without generating actual fault events and thus activating
already-configured spares; for example, zpool import'ing a pool
without all of its devices available does not necessarily cause faults
for the missing devices and thus cause them to be replaced with
spares. Instead ZFS will happily bring up the pool as non-redundant and
let the configured spares just sit there.
(In general, if ZFS spares are not being activated when you expect them to it's because whatever is going on is not generating spares-activating faults.)
This issue is one of the reasons that we are strongly considering completely ignoring ZFS's own spare support and writing our own system to handle it ourselves.
Sidebar: implementation differences between state-driven and event-driven
It might be tempting to argue that the event-driven approach is the right one because the state-driven approach is actually event driven under the hood; array state changes generate 'events' which the RAID code just handles magically instead of making all of them explicit and externally visible. This tempting argument is wrong, because there are other ways of handling state-driven spares than by explicitly handling specific state transitions (which are 'events' even if they are not coded as such). For example, one approach is to just call the 'do we need to activate a spare' code after every array state change, no matter what it is; this is sure to activate spares as soon as possible at the expense of running the logic more often than strictly necessary and making it have to deal some extra cases.