Wandering Thoughts archives

My view of Debian's behavior on package upgrades with new dependencies

In the primary Reddit discussion of my entry about actually learning apt and dpkg, yentity asked an entirely sensible question:

Why is this [apt-get's --with-new-pkgs] not enabled by default in debian / ubuntu ?

The ultimate answer here is 'because Debian has made a philosophical choice'. Specifically, Debian has decided that no matter what the person building the new version of a Debian package wants or feels is necessary, an 'apt-get upgrade' will never add additional packages to your system. If the builder of the package insists that a new version requires an additional package to be installed, it is better for the upgrade to not happen. Only 'apt-get install <pkg>' (or 'apt-get dist-upgrade') will ever add new packages to your system.

Regardless of what you think about its merits, this is a coherent position for Debian to take. In an anarchic packaging environment with individual Debian developers going their own way, it even has a fair amount of appeal. It certainly means that package maintainers have a strong pragmatic incentive not to add new dependencies, which probably serves to minimize it (which is one reason Debian has apt-get behave this way).

My personal view is that I prefer an environment where package builders are trusted to do the right thing with package dependencies in new versions of their packages, whatever that is. Packages can get new dependencies for all sorts of good reasons, including that what used to be a single package is being split up into several ones. As a sysadmin outsider, I'm not in a good position to second guess the package maintainer on what dependencies are right and whether or not a new one is harmful to my system, so in a trustworthy environment I'll just auto-install new dependencies (as we now do on Ubuntu where it's possible).

(The Debian package format has also made some structural decisions that make things like splitting packages more of a pain. In an RPM-based system, other packages often don't notice or care if you split yours up; in a Debian one, they're more likely to notice.)

It's worth pointing out that this trust fundamentally requires work and politics, in that it requires a policy on 'no unneeded dependencies' (and 'no surprises in package upgrades') and then a group of people who are empowered to judge and enforce the policy (overriding package maintainers when necessary). This sort of control probably does not go well with a relatively anarchic project and it's certainly a point of argument (and one could say that Debian already has enough of those).

An awkward confession and what we should do about it

I have an awkward confession.

At this point, we have been running Ubuntu machines for at least nine years or so, starting with Ubuntu 6.06 and moving forward from there. In all of that time, one of the things I haven't done (and I don't think we've done generally) is really dive in and learn about Debian packaging and package management. Oh sure, we can fiddle around with apt-get and a number of other superficial things, we've built modified preseeded install environments, and I've learned enough to modify existing Debian packages and rebuild them. But that's all. That leaves vast oceans of both dpkg and APT usage that we have barely touched, plus all of the additional tools and scripts around the Debian package ecosystem (some of which have been mentioned here by commentators).

I don't have a good explanation for why this has happened, and in particular why I haven't dug into Debian package (because diving into things is one of the things that I do). I can put together theories (including me not being entirely fond of Ubuntu even from the start), but it's all just speculation and if I'm honest it's post-facto excuses and rationalization.

But what it definitely is embarrassing and, in the long run, harmful. There are clearly things in the whole Debian package ecology that would improve our lives if we knew them (for example, I only recently discovered apt-get's --with-new-pkgs option). Yet what I can only describe as my stubborn refusal to dig into Debian packaging is keeping me from this stuff. I need to fix that. I don't necessarily need to know all of the advanced stuff (I may never create a Debian package from scratch), but I should at least understand the big picture and the details that matter to us.

(It should not be the case that I still know much more about RPM and yum/dnf than I do about the Debian equivalents.)

My goal is to be not necessarily an expert but at least honestly knowledgeable, both about the practical nuts and bolts operation of the system and about how everything works conceptually (including such perennial hot topics for me as the principles of the debconf system).

With all of that said, I have to admit that as yet I haven't figured out where I should start reading. Debian has a lot of documentation, but in the past my experience has been that much of it assumes a certain amount of initial context that I don't have yet. Possibly I should start by just reading through all of the APT and dpkg related manpages, trying to sort everything out, and keeping notes about things that I don't understand. Then I can seek further information.

(As is traditional on Wandering Thoughts, I'm writing this partly to spur myself into action.)

The sensible update for my vintage 2011 home machine

As it happens, I think that there is a sensible, boring answer to what I should do about my current home computer, one that bypasses all of my concerns over a potential replacement machine. It's just an answer I've been somewhat reluctant to do, partly because it's nowhere near as interesting as putting together a new machine and partly because it's going to require some awkward gyrations.

What I should really do is get a pair of 500 GB SSDs and use them to replace the current pair of system disks. A relatively small amount of space would go to an ext4 software RAID mirrored root filesystem (with all system stuff in it); the majority of it would go to a new ZFS pool where I would put my home directory and a few other filesystems where I want fast access (for example because I compile source code there). This would get me both ZFS and a speedup on a bunch of things that I do.

(At this point, some people are probably incredulous that I haven't already shifted my primary storage to SSDs.)

Following my experience with my work machine, I should probably also take a deep breath and switch my machine from 16 GB of RAM to 32 GB, because this will make ZFS happier. The RAM will not be reusable in a future machine, unlike the SSDs, and it's still surprisingly expensive, so it's tempting to skip this and see if I can make do (especially with ZFS on Linux ARC improvements).

While upgrading my entire machine is the attractive thing to do, at this point it seems very likely that the largest single practical speedup I could do is moving frequently used data to from HDs to SSDs. And I don't need a whole new machine to do that, by any means. There are limitations to what this will do for me (it won't speed up processing photos, for example), but I think it's likely to make a difference.

(Having looked at the CPU compatibility list for my Asus P8P67 LE motherboard, I don't think there's any point in trying to upgrade the CPU. In fact it basically seems like the LGA 1155 CPUs that my motherboard uses are now so old that they're basically legacy hardware that you pay extra for. Even if my perception here is wrong, it doesn't look like I can get much of a CPU upgrade.)

PS: My plan to put the root filesystem on the SSDs comes partly from necessity (it's currently on the HDs that the SSDs would replace) and partly because that's what I wound up doing on my work machine after contemplating my dilemma about this.

Sidebar: the irrational reason I've been avoiding doing things like this

For no particularly good reason, I've always found it easier to talk myself into buying a whole new machine every so often instead of progressively upgrading a machine piece by piece. As best I can figure it out, spending more money on something that's progressively becoming more and more obsolete just feels like a waste.

(As you may guess, I'm not one of the people who enjoys progressively upgrading machines piece by piece. In fact I think the only hardware changes I've done in machines to date has been replacing dead HDs.)

Wayland and graphics card uncertainty

Yesterday I wrote about several points of general technology churn that make me reluctant to think about a new PC. But as it happens there's an additional Linux specific issue that I worry about, and that's Wayland. More exactly, it's what Wayland is likely to require from graphics cards.

I use a very old fashioned X based environment, which means that all I need is old fashioned basic X. I think I run some things that like to see OpenGL, but probably not very many of them, and there's basic OpenGL support in even basic X these days. This has left me relatively indifferent to graphics cards and graphics card support levels; even what was a low end card at the time was (and is) good enough for my desktop.

I would like to keep on using my basic X environment for the lifetime of my next machine, but the forces behind Wayland are marching on with sufficient force that I don't think I can assume that any more. People are really trying to ship Wayland based desktops within the next couple of years on major Linux distributions (in particular, on Fedora), and once that happens I suspect that my sort of X environment will only have a few more years of viable life before toolkits and major programs basically stop supporting it.

(Oh, they'll technically 'support' X. But probably no one will be actively maintaining the code and so steadily increasing numbers of pieces will break.)

At that point, switching to Wayland will be non-optional for me (even if it results in upending my environment and makes me unhappy). In turn that means I'll need a graphics system that can handle Wayland well. Wayland is a modern system, so as far as I know it really wants things like hardware composition, hardware alpha-blending, and so on. Using Wayland without OpenGL-level hardware acceleration may be possible, but it's not likely to be pleasant.

What sort of graphics card (or integrated graphics) does this call for? I have no real idea, and I'm not sure anyone knows yet what you'll want to have for a good Wayland experience. That uncertainty makes me want to postpone buying a graphics card until we know more, which will probably need some brave Linux distribution to enable Wayland by default so that a lot of people run it.

(Of course I may be overthinking this as part of mostly not wanting to replace my current machine.)

How RPM handles configuration files, both technically and socially

In comments on my entry on Ubuntu packages playing 'hide the configuration file', James asked how RPM handles user editable configuration files (and then guaq gave a good answer). This is an important enough thing to talk about that I'm going to write an entry about it, because RPM's approach is both technical and social.

On the technical side, the first thing is that if the file hasn't changed between the old package and the new package update, RPM knows this and doesn't need to do anything; it doesn't touch the installed file and doesn't bother putting the stock version anywhere. If the configuration file has changed between the old and the new package and you've also edited the configuration file, most of the time the package will write out the new configuration file as <file>.rpmnew (for exact details of what happens when, see Jon Warbrick's writeup; while it's from some time ago, I don't believe RPM's behavior has changed since).

If this isn't enough, RPM packages can also do arbitrary things on package upgrades in their postinstall scripts. I believe that this has occasionally been used to do things like edit installed configuration files to modify or comment out configuration directives that have unavoidably changed or disappeared between software versions. However, this is where we shade into the social side of things.

Because RPM packages can't really try to do sophisticated things with configuration files, they have a strong social pressure to not need them. As a packager, part of your job is to make sure that the package's configuration files work right even within these restrictions. Generally you must keep existing configuration files working correctly even over package updates; resorting to pushing installed configuration files aside as .rpmsave files is strongly discouraged (as far as I know; it's certainly uncommon). This requires more up front work, is harder, and inevitably requires some tradeoffs (where people with edited configuration files don't magically get the benefit of new configuration options), but in my opinion it produces better long term benefits for sysadmins.

(It also encourages schemes to split configuration files up in various ways, because then it's more likely that most of the split configuration files will be unedited. RPM based systems are not alone in this, of course, and sometimes they don't do it as well as other systems.)

As a result, while you can do crazy things in RPM postinstall scripts, it's effectively discouraged. I would be surprised if Fedora packaging review was very receptive to mangling configuration files in postinstall scripts, for example (although it's not covered explicitly in the Fedora packaging guidelines). And of course the technical restriction that you can't ask the user any questions in your postinstall script limits the amount of things it's sensible to even try to have an ultra-smart script do.

This also affects what sort of default configuration it's sensible to set up in your package's stock files. Generally it's going to be better to create something minimal but stable instead of a complicated setup with lots of options that has to be managed by hand. Sure, many sysadmins will have to change your minimal setup, but on the other hand your package itself hopefully won't have to change and revise the setup from version to version. This keeps all of the changes, customizations, and updates in the hands of sysadmins, instead of making some of them your responsibility.

RPM's technical approach could not work without the social approach (although of course its technical limitations combined with sane people create the social approach). As a result, any RPM based system is going to have a terrible time with a program that needs configuration file customization and also changes its configuration files around in incompatible ways (either as for what's allowed in them or as to how they're organized). Fortunately there are not many programs like this, because they're unpopular with sysadmins even without RPM packaging issues.

Why it's irritating when Ubuntu packages don't include their configuration files

I'll start with my tweets:

This is my angry face when Ubuntu openssh-server generates a fixed /etc/ssh/sshd_config on the fly in postinst vs making it a config file.
Ubuntu dovecot also does the 'generate static configuration files at install time' thing and now I want to set it on fire.

(Twitter has become my venue of choice for short rants.)

Both of these packages have the same bad idea, although their mechanisms differ somewhat. Specifically, the installed and running software has some configuration files (here /etc/ssh/sshd_config and /etc/dovecot/conf.d/*), but these files are not actually packaged in the .debs; instead they are created at install time when the package is installed from scratch.

(The behavior may be there in the Debian versions as well, but I don't run Debian so I can't easily check.)

Years ago I wrote about wishing to be able to retrieve the stock version of some package file. At the time I gave a recipe for doing this by hand on a Debian based system; you apt-get download the .deb, use dpkg -x to unpack it in a temporary directory, and fish the file out. Naturally this only works if the .deb actually packages the file. If it does not, you have two options. First, you can find, read, and probably simulate by hand the process the .postinst script uses to create the file (the postinst script itself is likely in /var/lib/dpkg/info). Second, you have to install an uncustomized version of the package somehow, either on a test machine or by purging and then reinstalling the package (which is likely to be very disruptive if you have any important local customizations).

This is what I call 'not helpful'. There are all sorts of reasons to want to retrieve the original version of a configuration file that you've altered (or merely may have altered). Even when this is possible, how to do it is package specific, which means that you need package specific knowledge or research. Ubuntu packaging should do better here.

(I don't expect things to ever change, though. I rather expect that this is inherited from Debian and Debian has some intricate reason of their own for doing it this way (and for different packages doing it differently). Debian sometimes makes me angry with their apparent love of robot logic, technical intricacies, and apparent inability to ever do something in a single unified way.)

Apt-get and its irritating lack of easy selective upgrades

One of my many irritations with apt-get is that it doesn't easily allow you to only apply some of the pending updates. Sure, often you want to apply all of the updates (at least all of the unheld updates), but there are any number of cases where you want to be more selective. Sometimes you are in a rush and you want to apply only a few very urgent updates. Sometimes you want to apply updates in a specific order, updating some packages before others. Sometimes you want to apply most updates but temporarily exclude some that you consider low priority or disruptive.

With a package manager like yum (or now dnf) you can easily do either of these. If you just want to exclude some packages, you do that with '--exclude'; if you only want to upgrade some packages, you do that by supplying them as explicit arguments. And it's harmless to be a bit broad in your explicit arguments, because you're specifically only upgrading existing packages; you'll never install new ones out of nowhere.

apt-get does not support this usage as far as I can see. apt-get upgrade takes no package list and has no way of excluding some packages; it is an all or nothing operation, where the only way you have of being selective is to hold packages in advance in order to block their upgrades. In order to upgrade packages selectively, you must turn to 'apt-get install', probably with '--only-upgrade' so that you don't accidentally install new packages. And as far as I can tell this has no equivalent of yum's --exclude, so there's no easy way I can see of saying 'upgrade everything except the following packages'.

(apt-get install does at least support wildcards, or more exactly POSIX regular expressions. I don't know why they decided to use regular expressions instead of shell globbing, but it feels like a very Debian decision, especially the detail that it defaults to matching substrings.)

'apt-get install --only-upgrade PKG ...' solves about half of the problem (although clumsily) so I'm less disgruntled than I was at the start of writing this entry, but it's still not particularly great.

What happens when a modern Linux system boots without /bin/sh

So this happened to me:

It turns out that a whole lot of things explode when your system boots up with /bin/sh not working for some mysterious reason.

Here the mysterious reason was that there was an unresolved dynamic library symbol, so any attempt to run /bin/sh or /bin/bash died with an error message from the ELF interpreter.

The big surprise for me was just how far my systemd-based Fedora 23 machine managed to get despite this handicap. I certainly saw a cascade of unit failures in the startup messages so I knew that something bad had happened, but the first inkling I had of just how bad it was came when I tried to log in as root on the (text) console and the system just dumped me back at the login: prompt. Most of the system services had managed to start because their systemd .service files did not need /bin/sh to run; only a few things (some of them surprising) had failed, although a dependency chain for one of them wound up blocking the local resolving DNS server from starting.

The unpleasant surprise was how much depends on /bin/sh and /bin/bash working. I was able to log in as myself because I use a different shell, but obviously root was inaccessible, my own environment relies on a certain amount of shell scripts to be really functional, and a surprising number of standard binaries are shell scripts these days (/usr/bin/fgrep, for example). In the end I got somewhat lucky in that my regular account had sudo access and sudo can be used to run things directly, without needing /bin/sh or root's shell to be functioning.

(I mostly wound up using this to run less to read logs and eventually reboot. If I'd been thinking more carefully, I could have used sudo to run an alternate shell as root, which would have been almost as good as being able to log in directly.)

Another pretty useful thing here is how systemd captured a great deal of the error output from startup services and recorded it in the systemd journal. This gave me the exact error messages, for example, which is at least reassuring to have even if I don't understand what went wrong.

What I don't have here is an exciting story of how I revived a system despite its /bin/sh being broken. In the end the problem went away after I rebooted and then power cycled my workstation. Based on the symptoms I suspect that a page in RAM got scrambled somehow (which honestly is a bit unnerving).

As a side note, the most surprising thing that failed to start was udev trying to run the install command for the sound card drivers (specifically snd_pcm). I suspect that this is used to restore the sound volume settings to whatever they were the last time the system was shut down, but I don't know for sure because things didn't report the exact command being executed or whatever.

(My system has a 90-alsa-restore.rules udev rules file that tries to run alsactl. It's not clear to me if udev executes RUN+= commands via system(), which would have hit the issue, or in some more direct way. Maybe it depends on whether the RUN command seems to have anything that needs interpretation by the shell. I'm pretty certain that at least some udev RUN actions succeeded.)

Sidebar: What exactly was wrong

This was on my Fedora 23 office machine, where /bin/sh is bash, and bash was failing to start with a message to the effect of:

symbol lookup error: /bin/bash: undefined symbol: rl_unix_line_disc<binary garbage>

Bash does not mention a symbol with that exact name, but it does want to resolve and use rl_unix_line_discard. Interestingly, this is an internal symbol (it's both used and defined in bash); despite this, looking it up goes via the full dynamic linker symbol resolution process (as determined with the help of LD_DEBUG). My guess is that the end of the symbol name was overwritten in RAM with some garbage and that this probably happened in the Linux kernel page cache (since it kept reappearing with the same message, it can't have been in a per-process page).

Assuming I'm reading things correctly, the bytes of garbage are (in hex):

ae 37 d8 5f bf 6b d1 45 3a c0 d9 93 1b 44 12 2d 68 74

(less displays this as '<AE>7<D8>_<BF>k<D1>E:<C0>ٓ^R-ht', which doesn't fully capture it. I had to run a snippet of journalctl's raw output through 'od -t c -t x1' to get the exact hex.)