Why the Ubuntu package update process makes me irritated

February 21, 2016

We have fifty or so Ubuntu machines in our fleet, which means that when Ubuntu comes out with package updates we have to update them all. And this is rather a problem. First, obviously, fifty machines is too many to do updates on by hand; we need to automate this. Unfortunately Ubuntu (or Debian) has made a series of decisions about how packages and package updates work that cause significant amounts of pain here.

(Although Debian and Ubuntu share the same package format, I don't have enough experience with Debian to speak for it here. I wouldn't be surprised if much of this applied there, but I don't know for sure.)

To start with, Ubuntu package updates are slow. Well, okay, everyone's package updates are slow, but when you have fifty-odd machines the slowness means that doing them one by one can take a significant time. Even a modest package update can take an hour or so, which means that you'd really like to do them in parallel in some way.

(This does not bite too hard for normal updates, because you can do those once a day in the morning while you're getting coffee, reading email, and so on, and thus sort of don't care if they take a while. This really bites for urgent security updates that you want to apply right when you read Ubuntu's announcement emails, especially if those emails show up just before you're about to leave for the day.)

Next, Ubuntu package updates via apt-get are very verbose, and worse they're verbose in unpredictable ways, as packages feel free to natter about all sorts of things as they update. It may be nice to know that your package update is making progress, but without any regularity in the output it's extremely hard for a program to look through it and find problem indicators (and people have problems too). Other systems are much better here, because their normal output is very regular and you can immediately pick out deviations from that (either in a program or by eye). If you're going to automate a package update process in any way, you'd really like to be able to spot and report problems.

(Also, if you're updating a bunch of machines, it's a great thing to be able to notice problems on one machine before you go on to update another machine and have the same problems.)

In addition, Ubuntu packages are allowed to ask questions during package updates, which insures that some of them do (sometimes for trivia). A package update that pauses to ask you questions is not a package update that can be automated in a hands-off way. In fact it's even hard to automate in any way (at least well), because you need to build some way for a human to break into the process to answer those questions. Yes, in theory I think you can turn off these questions and force the package to take its default answer, but in practice I've felt for years that this is dangerous because nothing forces the default to be the right choice, or even sensible.

(I also expect that the verbosity of package updates would make it hard to spot times where a package wanted to ask a question but you forced it to take its default. These are cases where you really want to spot and review the decision, in case the package screwed up.)

The ideal hands off update process will never ask you questions and can be made to report only actual problems and what packages were updated. This makes it easy to do automatically and in parallel; you fire off a bunch of update processes on a bunch of machines, collect the output, and then can easily report both problems and actual packages updated. If you wanted to, you could use only limited parallelism, scan updates for problems, and freeze the whole process if any turn up. The only way that the Ubuntu package update process resembles this is that it's relatively easy to parse the output to determine what packages were updated.


Comments on this page:

By Anon at 2016-02-21 09:26:37:

You don't trust running unattended-upgrades (http://packages.ubuntu.com/wily/unattended-upgrades ) by hand (i.e. not via cron) because of neagtive expierences?

By John Morton at 2016-02-21 10:52:29:

I think you're overselling the problem of interactive questions during the update process. If a package, already installed, and being updated within a given release version, is asking for a configuration decision, that's a bug. This should simply never happen in main, and ought to be unheard of in universe and multiverse. Maybe PPAs might pull that sort of stunt, but you take your chances with those.

Of course, upgrades between releases are a different story; that pretty much always needs hand holding, and isn't something you want to do on production machines without a planned outage.

Regarding, defaults, you seem to have a no win situation. Either packages have questions that force interactivity, or they have defaults that could be wrong. However every possible configuration value that doesn't have an interactive question has a tacit default that could be wrong for your situation.

(Thoroughly agree that apt/dpkg is good at interactive, but bad at log-and-analyse/ A pity.)

By Jesse D at 2016-02-21 11:29:41:

Have you looked into Spacewalk, the community version of Red Hat Satellite? It supposedly can manage Ubuntu based systems. No idea how it handles interactive installs.

http://spacewalk.redhat.com/

By Ewen McNeill at 2016-02-21 14:07:12:

I've been told by a Debian Developer that packages asking unskippable questions during install is a bug, that should be reported -- the package should be asking via something like DebConf which allows answers to be pre-seeded and/or defaulted. Which means for N systems with a package installed, at worst you should only need to do it interactively for the major variations -- the rest can have the answer pre-seeded.

Of course as someone else said, packages being updated (ie, patched) within a given release cycle should not be asking new questions, and off hand I don't recall (m)any that do. (Config file changes is the main one, but those don't normally get changed within a release cycle by, eg, security patches.)

Finally "apt-get --download-only upgrade" can be useful to automate to pre-stage all the package files needed onto the system, even if you're going to apply the updates by hand. Typically I find even with a fast mirror, that download step takes half the time.

Ewen

PS: I use a tool that reports packages/versions on my systems back to a central point as the main way of auditing that packages did get updated. Rather than trying to parse the combined apt/dpkg/package-echo output.

By cks at 2016-02-21 15:32:18:

I know for sure that updates to grub have asked us questions about our grub configuration. It's a routine occurrence to the point where we now hold grub to avoid it. I believe that mdadm has had similar issues (I know we routinely hold it and consider upgrading it a problem). I'm pretty sure we've seen other packages do this every so often.

As for this being a bug; sure, I'm willing to accept that Debian and maybe even Ubuntu considers it a bug. But it clearly keeps happening, which means that we have to expect it to and build an update process around that assumption.

I don't think we've ever looked at unattended-upgrades. Having taken a quick look at eg the Debian wiki page on it, it's not clear to me what running it by hand gives us over doing updates with apt-get. There may be advantages, but if so the information isn't selling them (and there are clearly some drawbacks, like needing a separate command to see what got updated).

I don't think we're interested in Spacewalk; for various reasons, including those questions, we want to directly control and supervise the update process, not have it just happen magically. Perhaps this can be done with Spacewalk, but if so it seems likely we'd have to set up a lot of infrastructure just to get that.

Ewen's suggestion may turn out to be a good idea. I'll have to time how long package downloads take; while my perception is that they're fast (partly because we maintain our own local mirror that has many of them), but I may be wrong. If it's actually slower than I'm expecting, downloading them in advance in parallel is an easy speedup. Perhaps we should do this no matter what, because it probably can't hurt.

(I assume that if a download-only bit fails to fetch some packages, a later 'apt-get upgrade' will try to re-fetch them. In any case I think package download failures are rare for us.)

Debian, and all Debian-based variants that use apt-get and dpkg, have ways of automating everything. We're a small'ish shop. That is to say, I manage close to 100 machines single-handedly, all running Ubuntu 14.04. But the cool thing about it is that it doesn't matter if I manage 100 or 100,000 -- it's all the same to me.

Scripts are your friend:

   run_apt () {
   	echo
   	echo "+---"
   	export DEBIAN_FRONTEND="noninteractive"
   	/usr/bin/debconf-set-selections << EOF
   debconf	debconf/priority	select	critical
   debconf	debconf/frontend	select	Noninteractive
   EOF
   	apt-get \
   		--quiet \
   		--option DPkg::Options::="--force-overwrite" \
   		--option DPkg::Options::="--force-confnew" \
   		--assume-yes \
   		--fix-missing \
   		$*
       rc=$?
   	/usr/bin/debconf-set-selections << EOF
   debconf	debconf/priority	select	high
   debconf	debconf/frontend	select	Dialog
   EOF
   	unset DEBIAN_FRONTEND
   	echo "+---"
       echo "Exit code: ${rc}"
   	echo
       return ${rc}
   }

   #
   # Examples:
   #
   # run_apt upgrade
   # run_apt dist-upgrade
   # run_apt install some-package

Example output:

   root@figment:~# run_apt update
   +---
   Ign http://dl.google.com stable InRelease
   Hit http://repo.steampowered.com precise InRelease
   [...<boring stuff deleted>...]
   Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
   Fetched 3,157 kB in 13s (237 kB/s)
   Reading package lists...
   +---
   Exit code: 0
   root@figment:~# run_apt --reinstall install bar
   +---
   Reading package lists...
   Building dependency tree...
   Reading state information...
   The following package was automatically installed and is no longer required:
     libvncclient0
   Use 'apt-get autoremove' to remove it.
   0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
   Need to get 41.7 kB of archives.
   After this operation, 0 B of additional disk space will be used.
   Get:1 http://us.archive.ubuntu.com/ubuntu/ trusty/universe bar amd64 1.11.1-2 [41.7 kB]
   Fetched 41.7 kB in 0s (143 kB/s)
   (Reading database ... 1289281 files and directories currently installed.)
   Preparing to unpack .../bar_1.11.1-2_amd64.deb ...
   Unpacking bar (1.11.1-2) over (1.11.1-2) ...
   Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
   Setting up bar (1.11.1-2) ...
   +---
   Exit code: 0
   root@figment:~# 

This little snippet of code is part of a larger script that I use to ssh into every machine I manage and install updates in parallel.

Now that you have the power in your hands, go forth, my friend, and code... Code like the wind!

By cks at 2016-02-22 00:24:50:

Thanks for sharing this script. I suspect that the options here will deal with at least some of our issues, although there's still the 'is telling a question to take the default answer going to blow up in our faces' issue.

(Having such a thing happen would presumably be a bug, but that's small comfort when you have a fleet of damaged machines. On the other hand, if people have had long-term, problem free success with this approach, which it sounds like you have, then we're unlikely to run into problems; we don't have any particularly weird packages or the like.)

My partial solution to this problem is the apt-dater (http://packages.ubuntu.com/apt-dater) package. I can test out upgrades on one host, and if everything runs cleanly I can do updates in parallel on a set of other machines.

Error detection is not perfect (there are false positives) but apt-dater tries to catch some error messages.

By Алексей at 2016-06-04 04:15:05:

Regarding slow updates. If your power supply is reliable enough, you may consider using dpkg option --unsafe-io or like that (don't remember exact name, see man dpkg).

Speaking about packages asking questions, we've never experienced that during normal upgrades in 5+ years using Debian stable. Don't know about Ubuntu though, we've got rid of the last Ubuntu server years ago.

Written on 21 February 2016.
« My two usage cases for Let's Encrypt certificates
The university's coordination problem »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun Feb 21 04:07:54 2016
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.