2009-05-05
How we periodically scrub our ZFS pools
The problem with the simple approach to scrubbing your ZFS pools (just
do 'zpool scrub ...' every so often) is that ZFS pool scrubs put
enough of a load on our systems that we don't want to do them during
the week and we don't want to do more than one of them at once (well,
not more than one per fileserver). And we certainly don't want to have
to manage the whole process by hand. So recently I wrote a script to
automate the process.
The script's job is to scrub pools one by one during the weekend, if they haven't been scrubbed too recently and they're healthy. To tell if pools have been scrubbed recently, we keep a flag file in the root filesystem of the pool; the modification time of the file is when we kicked off the last scrub.
(As it happens, we don't use the root filesystem of our pools for anything and they're always mounted in a consistent place, so the flag file isn't disturbing anything and it's easy to find.)
The script gets started from cron early on Saturday morning and then
runs in the background repeatedly starting a pool scrub and waiting for
it to finish. In the Unix tradition of clubbing problems with existing
programs, it uses find on all of the flag files to find out which flag
files are old enough that their pools are candidates for scrubbing, and
then ls to order them from oldest to newest so that it can find the
oldest healthy pool. Waiting for pool scrubs to finish is done the brute
force way; the script repeatedly runs 'zpool status' and waits until
there are no 'scrub:' lines that indicate ongoing scrubs or resilvers.
(Except not. Because I am paranoid, it works the other way around;
it throws away all 'scrub:' lines that it knows are good, and if
there's anything left it assumes that a pool is still scrubbing or
resilvering. This overcaution
may cause us problems someday.)
The script exits when there are no pools left to scrub or if is after its exit time, currently Monday at 1am. (This doesn't quite mean that pool scrubbing will stop at Monday at 1am; it means that no pool scrubs will start after that point. Our biggest pools scrub in six and a half hours currently, so even in the worst case we should be done before 8am Monday.)