Wandering Thoughts archives

2010-07-21

The easy way to do fast OS upgrades

We recently went through the experience of upgrading all of our ZFS fileservers from Solaris 10 update 6 to Solaris 10 update 8. This took somewhere around twenty minutes of downtime per fileserver, most of which was waiting for ZFS pools to slowly import.

You might wonder how we got an OS upgrade to go so fast. The answer is that we cheated, twice.

The first way we cheated is that we didn't upgrade the OS; instead, we (re)installed Solaris 10 update 8 from scratch. This is our traditional approach with most of our servers (anything that doesn't have important local data, and we try not to have servers with important local data). We need to be able to reinstall servers anyways to cope with hardware problems, and once you have a well-tested reinstall process you might as well use it for everything.

The second way we cheated is that we didn't reinstall S10U8 on the same machine. Our ZFS fileservers have swappable disks, so we did the install on a spare server (with identical hardware) then swapped the new S10U8 disks into the actual physical fileserver during the downtime. And then, of course, we had to fix up all of the places on the system that knew what host it was and what hardware it was running on, which is really why the downtimes took more than a minute or two.

(This also gave us a rapid fallback if we had to; we could have just pulled the S10U8 disks and put the S10U6 disks back in.)

Now, various OSes have various sorts of software based fast upgrade schemes, and some of them even work reliably. But you can be pretty sure that swapping disks will work for anything, provided only that you can rename a system and move its system disks between hardware units, and you're going to want to work out how to do both of those anyways.

(Sadly, these days systems are increasingly welded to the hardware that they were installed on in various perverse ways that require annoying amounts of effort to reverse or override.)

sysadmin/FastOSUpgrades written at 00:22:26; Add Comment


Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.