The balance of power between distributions and software authors

March 1, 2021

In a comment on my entry on how modern software controls dependencies because it helps software authors, Peter Donis raised a good and interesting point:

I understand the issue you are describing for software authors, but as a software user, if bundling static dependencies instead of using shared libraries means security vulnerabilities in software I'm using take longer to get fixed, then I'll just have to stop using that software altogether. And for the vast majority of software I use, I don't get it directly from the author; I get it from my distro, so what matters to me is how easily the distro can fix issues and push updates to me.

And the distro maintainers know that, and they know that users like me are depending on the distros to get those vulnerabilities fixed and push updates in a timely manner; so if particular software programs make it hard for them to do that, they have an incentive to stop packaging those programs. It seems to me that, if push comes to shove, the distros have the upper hand here. [...]

My view is that there is a somewhat complicated balance of power involved between software authors and distributions, all starting with the people who are actually using the software. This balance of power used to be strongly with distributions, but various aspects of modern software has shifted it toward software authors.

As a person using software, I want to, well, use the software, with as little hassle and as few security issues, bugs, and other problems as possible. Like most people I'm broadly indifferent to how this is done, because my goal is to use the software. However, how much aggravation people will put up with to use software depends on how important it is to them, and I'm no exception to this.

(This balance is individual, not universal; different people and organizations have different needs and priorities.)

If the software is not all that critical or important to me, my threshold for aggravation is low. In fact if the aggravation level is too high I may not even use the software. Here the distribution mostly holds the balance of power, as in Peter Donis' comment, because obtaining software through the distribution is often much easier (although that's changing). If the distribution refuses to package a particular piece of software and it's not that important, I may well do without; if it only packages an old version, that version can be good enough. Software authors may need to play along with what the distributions want or be left out in the cold.

But if the software is important or critical to us, our threshold of aggravation is very high. Doing without or living with bugs fixed upstream in more recent versions becomes less and less of an option or is outright off the table. If we can use a distribution version we generally will, because that's less work, but if we have to we will abandon the distribution and work directly with the upstream version in one way or another. Here the software author holds the balance of power. The distributions can either go along with how the software author is working or be bypassed entirely.

(As an illustration of this power, we install the upstream versions of Prometheus and rspamd instead of the Ubuntu versions. And we use Grafana despite it not being packaged in Ubuntu 18.04.)

This means that one of factors in the balance of power is what sort of software it is, which goes a long way to determining how critical it is to people. The authors of a utility program usually have a lot less leverage than the authors of important services. Authors of important services are also much more exposed to people's anger if distributions make mistakes, as they do every so often, which gives these software authors some biases.

Modern software ecologies like Rust, Python, NPM, and Go have shifted this balance of power in an important way, especially for things like utility programs, because all of them provide a simple way for people to install a program straight from its authors, bypassing the distribution. I don't think that the ecologies are deliberately trying to make distributions obsolete, but they do have a strong motivation to remove the distributions as gatekeepers. No distribution will ever package all of the software that's written in any language, so the easier the language makes it to install software outside of a distribution, the more software written in it will be used and spread.

(These ecologies do somewhat less well at letting you easily keep the programs you've installed up to date, but I suspect that that will come with time. Also, pragmatically a lot of people don't really worry about that for a lot of programs, especially utility programs.)

Sidebar: The historical balance of power shift

In the old days, it was often difficult (and time consuming) to either compile your own version of something written in C (or C++) or obtain a working binary from upstream. For building, there was no standard build system, you often had to pick a bunch of install time options, and you might need to do this for a whole collection of dependencies (which you would need to install somewhere that building the main program could find). For binaries, you had to deal with shared library versions and ABI issues. Distributions did all of this work for you, and it was hard work.

Modern software environments like Rust, Go, NPM, and Python change both the binary and the build your own side of this. Rust and Go statically link everything by default and produce binaries that are often universal across, eg, Linux on 64-bit x86, making it basically trouble free to get the upstream one. For building your own, all of them have single commands that fetch and build most programs for you, including the right versions of all of their dependencies.

Comments on this page:

Since a comment of mine prompted this post, I'll offer a couple of thoughts.

First, I think there is a key difference between individual users, like me, and organizations, like the one cks works for. As an individual user, the likelihood of there being an application that (a) is important enough that I will do whatever it takes to get timely updates, and (b) isn't packaged and updated in a timely manner by my distro, is pretty small. (For me, personally, the number of such applications is zero, and has been so for as long as I've been using Linux, which is almost 20 years now.) But that's because I'm just one user. For an organization that has to support lots of users, who are likely to have widely varying needs and preferences, the likelihood of there being such applications is much higher. (cks names three for his organization in the post.) Also, an organization has more resources available to do the work involved in using a version other than the one packaged with the distro, so their threshold for how important an application has to be for them to do that is lower than mine as an individual user.

Second, the software ecologies cks mentions, while I agree they certainly would like to displace distros as gatekeepers, can only do that if they become gatekeepers themselves. Currently they're not. The ecology I'm most familiar with is Python; yes, I can pip install any Python project that's on PyPI (which includes any Python project anyone would likely be interested in), and I might want to do that even if my distro packages the project since their version is likely out of date. (Pip itself has this same problem, as every Python user who has seen the annoying messages telling you you should update your pip version can attest.) But no project owner on PyPI is going to give me any kind of assurance that nothing will break on my system when I pip install their project. And of course the risk involved here grows exponentially as the number of dependencies in the project goes up (Python projects seem to be fairly mild in this respect as compared with, say, npm). Whereas, if I install a Python project from my distro, I at least know that the distro maintainers have tested it with that distro's particular combination of all other stuff, including all the project's dependencies, and it didn't break.

Written on 01 March 2021.
« Dot-separated DNS name components aren't even necessarily subdomains, illustrated
Link: Taking This Serially »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Mar 1 23:01:21 2021
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.