Why you mostly don't want to do in-place Linux version upgrades

April 25, 2016

I mentioned yesterday that we don't do in-place distribution upgrades, eg to go from Ubuntu 12.04 to 14.04; instead we rebuild starting from scratch. It's my view that in-place upgrades of at least common Linux distributions are often a bad idea for a server fleet even when they're supported. I have three reasons for this, in order of increasing importance.

First, an in place upgrade generally involves more service downtime or at least instability than a server swap. In-place upgrades generally take some time (possibly in the hours range), during which things may be at least a little bit unstable as core portions of the system are swapped around (such as core shared libraries, Apache and MySQL/PostgreSQL installs, the mailer, your IMAP server, and so on). A server swap is a few minutes of downtime and you're done.

Second, it's undeniable that an in-place upgrade is a bit more risky than a server replacement. With a server replacement you can build and test the replacement in advance, and you also can revert back to the old version of the server if there are problems with the new one (which we've had to do a few times). For most Linux servers, an in place OS upgrade is a one way thing that's hard to test.

(In theory you can test it by rebuilding an exact duplicate of your current server and then running it through an in-place upgrade, but if you're going to go to that much more work why not just build a new server to start with?)

But those are relatively small reasons. The big reason to rebuild from scratch is that an OS version change means that it's time to re-evaluate whether what you were customizing on the old OS still needs to be done, if you're doing it the right way, and if you now need additional customizations because of new things on the OS. Or, for that matter, because your own environment has changed and some thing you were reflexively doing is now pointless or wrong. Sometimes this is an obvious need, such as Ubuntu's shift from Upstart in 14.04 LTS to systemd in 16.04, but often it can be more subtle than that. Do you still need that sysctl setting, that kernel module blacklist, or that bug workaround, or has the new release made it obsolete?

Again, in theory you can look into this (and prepare new configuration files for new versions of software) by building out a test server before you do in-place upgrades of your existing fleet. In practice I think it's much easier to do this well and to have everything properly prepared if you start from scratch with the new version. Starting from scratch gives you a totally clean slate where you can carefully track and verify every change you do to a stock install.

Of course all of this assumes that you have spare servers that you can use for this. You may not for various reasons, and in that case an in-place upgrade can be the best option in practice despite everything I've written. And when it is your best option, it's great if your Linux (or other OS) actively supports it (Debian and I believe Ubuntu), as opposed to grudging support (Fedora) or no support at all (RHEL/CentOS).


Comments on this page:

By Ewen McNeill at 2016-04-25 02:45:01:

Yes, Debian and Ubuntu do both support in-place upgrades, at least to "next stable release" (and in the case of Ubuntu, "LTS to next LTS"). I've had good success with both, although it does definitely help to have rehearsed it, or have time to tidy up after the upgrade (usually 30-60 minutes).

Without disagreeing with what you say, typically I'm happier to do in place upgrades of virtual machines where I can easily snapshot a "known good" before state and roll back if the upgrade goes badly astray (and also,eg rehearse with a clone of the VM). I will upgrade physical hardware too, but typically only where (a) I know the upgrade process to that version well, (b) it can live with, say, 60 minutes of partial service (eg, few users), and (c) there isn't spare hardware around.

Ewen

By Joel Dinel at 2016-04-25 07:53:17:

While I agree with Ewen's comment, I recently took the same stance as you and do server rebuilds on new releases. Even if my "servers" are really personal VPS, rebuilding lets me test my Ansible deployment recipes. This often leads to slimmer builds with less tweaks, and optimization of my Ansible recipes as I am definitely still learning it.

Deploying a temporary second VPS "on the side" while I rebuild is also extremely cheap, usually running me less than a dollar.

- Joel Dinel

By Tim Evans at 2016-04-25 08:51:21:

In this context, I was a fan of Solaris' 'Live Upgrade,' which, among other things, allowed an upgrade to be applied to an identical 'alternate boot environment,' while the system ran from the prior stable one. Once the upgrade was done, rebooting from the upgraded 'boot environment' was both quick and quickly reversible. -- Tim Evans, tkevans@tkevans.com

Written on 25 April 2016.
« Why we have CentOS machines as well as Ubuntu ones
Bad slide navigation on the web and understanding why it's bad »

Page tools: View Source, View Normal.
Search:
Login: Password:

Last modified: Mon Apr 25 01:40:39 2016
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.