We have fifty or so Ubuntu machines in our fleet, which means that when Ubuntu comes out with package updates we have to update them all. And this is rather a problem. First, obviously, fifty machines is too many to do updates on by hand; we need to automate this. Unfortunately Ubuntu (or Debian) has made a series of decisions about how packages and package updates work that cause significant amounts of pain here.

(Although Debian and Ubuntu share the same package format, I don't have enough experience with Debian to speak for it here. I wouldn't be surprised if much of this applied there, but I don't know for sure.)

To start with, Ubuntu package updates are slow. Well, okay, everyone's package updates are slow, but when you have fifty-odd machines the slowness means that doing them one by one can take a significant time. Even a modest package update can take an hour or so, which means that you'd really like to do them in parallel in some way.

(This does not bite too hard for normal updates, because you can do those once a day in the morning while you're getting coffee, reading email, and so on, and thus sort of don't care if they take a while. This really bites for urgent security updates that you want to apply right when you read Ubuntu's announcement emails, especially if those emails show up just before you're about to leave for the day.)

Next, Ubuntu package updates via apt-get are very verbose, and worse they're verbose in unpredictable ways, as packages feel free to natter about all sorts of things as they update. It may be nice to know that your package update is making progress, but without any regularity in the output it's extremely hard for a program to look through it and find problem indicators (and people have problems too). Other systems are much better here, because their normal output is very regular and you can immediately pick out deviations from that (either in a program or by eye). If you're going to automate a package update process in any way, you'd really like to be able to spot and report problems.

(Also, if you're updating a bunch of machines, it's a great thing to be able to notice problems on one machine before you go on to update another machine and have the same problems.)

In addition, Ubuntu packages are allowed to ask questions during package updates, which insures that some of them do (sometimes for trivia). A package update that pauses to ask you questions is not a package update that can be automated in a hands-off way. In fact it's even hard to automate in any way (at least well), because you need to build some way for a human to break into the process to answer those questions. Yes, in theory I think you can turn off these questions and force the package to take its default answer, but in practice I've felt for years that this is dangerous because nothing forces the default to be the right choice, or even sensible.

(I also expect that the verbosity of package updates would make it hard to spot times where a package wanted to ask a question but you forced it to take its default. These are cases where you really want to spot and review the decision, in case the package screwed up.)

The ideal hands off update process will never ask you questions and can be made to report only actual problems and what packages were updated. This makes it easy to do automatically and in parallel; you fire off a bunch of update processes on a bunch of machines, collect the output, and then can easily report both problems and actual packages updated. If you wanted to, you could use only limited parallelism, scan updates for problems, and freeze the whole process if any turn up. The only way that the Ubuntu package update process resembles this is that it's relatively easy to parse the output to determine what packages were updated.