2013-12-26
How ZFS scrubs routinely save us
A while back I wrote about how ZFS resilvering saved us and mentioned in passing that there are a number of ways that ZFS routinely saves us in the small. Today I want to talk about one of them, namely ZFS scrubs.
Put simply, ZFS scrubs are integrity checks of your ZFS pools. When you scrub a pool it checks and verifies all copies of all pool data to make sure that they're fully intact. When it finds a checksum inconsistency it will repair it; if things are really bad and it's not possible to repair it, it'll tell you what got damaged so you can restore it from backups. If a scrub discovers a read error it generally won't try to rewrite the data but it will at least tell you about it.
We regularly scrub our pools through automation. This periodically turns up transient checksum errors, which it also fixes. So this is the first little save; ZFS has detected and fixed potential data problems for us and it does it on an essentially ongoing basis. As a pragmatic thing the scrubs also check for read errors (although they can't fix them) and so give us early warning on disks we probably want to replace. They also give us a way to check if read errors are transient or permanent; we simply schedule a scrub and see if the scrub gets errors.
(A surprisingly large amount of the time the scrub does not, either because the error was genuinely transient or because whatever object was using the bad sector has been deleted since then.)
As a corollary, forcing an immediate scrub lets us find out if there are any latent problems (which can have many potential causes). It's routine for us to force scrubs after significant outage events, such as an iSCSI backend losing power, to make sure that no data got lost in the chaos.
Of course it would be better if we didn't have checksum errors happen in the first place. But given that we have something going wrong, I'd much rather know about it and have it get fixed than not. ZFS does this for us without fuss or hassle, and that routinely saves us in the small.
(Much of this can be done by any RAID system with routine RAID array scans; Linux software RAID can do this, for example, and is often configured to do it. What is different about ZFS is that ZFS can tell which copy of inconsistent data is correct and which isn't. Other RAID systems have to just guess.)
(I've talked about the overall purposes of ZFS scrubs in an aside here.)
2013-12-18
Thinking about what we'll need for reproducible OmniOS installs
We have a thing here where we like all of our fileservers to be exactly identical to each other (apart from configuration information), even if they happen to be installed or reinstalled some time apart. In the old days of basically static systems this was easy to achieve; if you installed everything from the media for OS release X.Y, they all wound up the same. Today, in a world of (online) patches and package updates and security fixes and so on, this is not necessarily all that simple. This is especially so for free software whose distribution mirrors may not have either the disk space or the interest in keeping every package version ever released around for perpetuity (especially if those package versions are perhaps known to have security bugs).
(This generously assumes that the package management system can be used to install a very specific version of a set of packages even when there are newer versions available. Not all package systems are happy about that.)
OmniOS is installed from versioned install media and then updated and supplemented through package updates with IPS. In practice the OmniOS base package repository is not enough and we'll likely supplement it with pkgsrc to get various important things. IPS supports mirroring selected packages to your own local repository (although I don't know if this can easily be done on a non-Solaris platform). I haven't looked at pkgsrc but it can probably do something.
There is also two somewhat crazier options that I've thought of: building our own install media with current updates and packages rolled in, and ZFS level copies. Building our own install media would cut out a bunch of the update process although not all of it, since pkgsrc probably couldn't be bundled into the install media. As a result I probably won't look much at this. ZFS level copies are much more potentially promising.
OmniOS installs to a ZFS root pool, in which you have a core filesystem.
OmniOS already supports multiple iterations of this core filesystem
through ZFS snapshots, clones, and so on; it uses this to support having
multiple boot environments for rollback purposes. So what would happen
if we installed the base OmniOS and then instead of customizing it,
simply copied over the current root filesystem from a working system
with zfs send and zfs recv, modified or reset things like IP address
information, and then made this copy the current boot environment? This
could blow up spectacularly or it could be a great way to make an exact
duplicate of an existing system.
(One issue is actually finding all of the places where an OmniOS system
stashes things like IP addresses and other host-specific bits. In the
new world of state-manipulating commands like dladm and ipadm, they
are probably not stored in any simple text file. Another issue is making
this process play nice with the boot environment maintenance commands.)
Sadly the whole ZFS idea is probably working too hard at something we can solve in simpler, more brute force ways (at the cost of some more disk space). I'm not sure if it's a good use of my time to investigate it much.
(One excuse is that I should understand boot environment administration and so on much better than I currently do, and that all by itself would tell me a lot.)