Modern software controls dependencies because it helps software authors

February 20, 2021

Over on Twitter I had a hot take:

Hot take: Every distribution packager who's saying "you shouldn't bundle dependencies" is implicitly telling software authors "you should do more work for us and limit the features (and reliability) of your software".

(This was sparked by reading The modern packager’s security nightmare, via. I'm not exactly on one side or the other, but I do think distributions should be honest about what they're asking for and I don't think they're going to get it.)

This hot take is a bit too narrow. What really matters is software authors and modern software systems restricting the versions of dependencies (for both maximum and minimum versions). Explicit or implicit bundling on top of that just makes the problem slightly worse for distributions.

For software authors, restricting the versions of dependencies that they work with reduces the amount of work that they have to do, both to test against a range of versions and to either forever chase after whatever changes those dependencies like to make or to forever limit what features of dependencies they use to ones available in old versions (and sometimes both at once). In theory, both testing and chasing after changes would be dealt with by Semantic Versioning (if everyone followed it), at least for a single program. In practice, not only are people fallible but also people have a different understanding of what semantic versioning means because semantic versioning is ultimately a social thing, not a technical one. Our field's history has shown (sometimes vividly) that if software authors allow versions of dependencies to move on them, soon or later things break and the software author has to fix it.

(There's also the practical issue that not all dependencies even claim or agree to follow semantic versioning in the first place.)

For distributors, once software authors start restricting versions the distributor has both an upgrade problem and a distribution problem. On the upgrade side, dealing with an issue in a program may now require persuading it to accept a new version of a dependency. On the distribution side, it's now likely that you'll have multiple programs that have different version requirements for the same dependency. At the very least this multiplies the packages involved.

(Many distributions also have package system design problems that restricts the range of versions of things that they can have installed at the same time. Even under Semantic Versioning, this is a problem for the distribution the moment that you have two programs with conflicting version requirements that can't both be packaged and installed at the same time.)

However, there's no free lunch here. What distributors want when they ask for unbundled dependencies without version restrictions is for software authors to do the work to accept any version of their dependencies, or at least any version that falls within Semantic Versioning, and for dependencies to faithfully follow semver and also make it possible to package and install different major versions (at least) at the same time. Accepting a broad version range of your dependencies is actual work, even apart from the limitations it may impose on what your code and your software can do. Software authors and the creators of software package ecosystems (like Go and Rust) are not refusing to do this because they don't like distributions; they are refusing to do this because they have found, over and over again, that this doesn't really work and does cause problems for software authors (and often users of programs) in the long run.

(The software community that's gone through this experience the most visibly is Go, which started out with intrinsically unversioned dependencies that were used universally across all your programs by default and wound up switching to strongly versioned dependencies after many people had many problems with that initial state. Go experienced so many problems that they adopted an unusually strict and software author friendly versioning scheme.)

It's popular for people to argue that software authors should be doing this work anyway even if the distributions weren't asking for it, so them actually doing it is no big deal. This is quite convenient for the people making the argument, but it doesn't make the argument valid. Software authors don't owe anyone any work what so ever; they do whatever work serves their needs and is interesting to them. With limited time and interest, it's both rational and proper for software authors to optimize for their own development.

PS: Generally distributions also want some combination of all software to update to the latest version of their dependencies and for dependencies to explicitly support older versions. This is also extra work for software authors, especially when the distribution also wants it to happen for older versions of programs.

Comments on this page:

I understand the issue you are describing for software authors, but as a software user, if bundling static dependencies instead of using shared libraries means security vulnerabilities in software I'm using take longer to get fixed, then I'll just have to stop using that software altogether. And for the vast majority of software I use, I don't get it directly from the author; I get it from my distro, so what matters to me is how easily the distro can fix issues and push updates to me.

And the distro maintainers know that, and they know that users like me are depending on the distros to get those vulnerabilities fixed and push updates in a timely manner; so if particular software programs make it hard for them to do that, they have an incentive to stop packaging those programs. It seems to me that, if push comes to shove, the distros have the upper hand here. Sure, particular communities like the Go community, that have a large corporate sponsor behind them, might be able to push back; but most individual software authors won't have that leverage.

By @ch2500 at 2021-02-22 17:29:13:

This is a point that I should be writing into a blog post somewhere, but:

If application authors want and need this deep level of control over their dependencies, this shakes up the definition of distributions. For a while, it was quite clear that a distribution picks applications and libraries, and tries to form a somewhat coherent deliverable. That included common ways of installing (rpm/apt) and configuring the applications ("all services should work after installations"), sometimes their look (desktop themes). This meant that all packages were treated mostly equally.

Now, when each application bundles (most of) its dependencies, there is very little value distributions add to that. Basically, users should (and already do!) just download the upstream binaries and run them.

To me it looks like, with this big bundling debate, somewhat spawned by Docker, Snap, etc., we are going back to the old Vendor OS model: you get an installer from your OS vendor (previously called distribution), which happens to be a kernel, an init system, a C compiler, and the rest is going to be "your problem".

A large part of what we all got from distributions was: easy installation, (mostly not noticeable) security updates, and mostly trustworthy binaries. I am not looking forward to a time when, places that need to care about that, have to rebuild everything they run on their own - because software will not be in distributions anymore, and (re)building will become harder over time (and unsupported by upstreams).

By John Wiersba at 2021-03-02 17:00:01:

A large part of what we all got from distributions was: easy installation, (mostly not noticeable) security updates, and mostly trustworthy binaries.

^ This

  1. Easy installation, including pre-built and pre-configured binaries, and a working, user-friendly update system (c.f. Windows or Mac)
  2. Trustworthy, curated, and vetted applications and tools

I think you've hit the nail on the head for me and many software users. Very rarely do I install software from outside my distribution's packaging system, not only because I want it to just work, but also because I don't want to have to vet the application and its author.

As you mention in your more recent post I'm sometimes willing to go outside the distribution's packaging system, but only when the particular application/tool and/or it's age are highly important to me.

As a software developer, the modern trend is to use the language's own ecosystem for distribution, but I think that loses a lot from #2 above. That also goes for ecologies like Android's Play Store, Firefox's Add-Ons, snaps and docker-based applications.

The first thing I look for in a distribution is trustworthiness. But in general, I don't get the warm fuzzies from any of the above ecosystems.

Written on 20 February 2021.
« ZFS pool partial (selective) feature upgrades are coming in OpenZFS
How I set up testing alerts in our Prometheus environment »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sat Feb 20 21:53:02 2021
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.