Wandering Thoughts archives

2021-09-18

One major obstacle to unifying the two types of package managers

One common reaction on lobste.rs to my entry on how there are two types of package managers was to hope that the two types are unified somehow, or that people can work toward unifying them. Unfortunately, my view is that this currently has a major technical obstacle that we don't have good solutions for, which is the handling of multiple versions of dependencies.

A major difference between what I called program managers (such as Debian's apt) and module managers (such as Python's Pip) is their handling or non-handling of multiple versions of dependencies. Program managers are built with the general assumption of a single (global) version of each dependency that will be used by everything that uses it, while module managers allow each top level entity you use them on (program, software module, etc) to have different versions of its dependencies.

You can imagine a system where a module manager (like pip) hooks into a program manager to install a package globally, or a program manager (like apt) automatically also installs packages from a language source like PyPI. But any simple system like this goes off the rails the moment you have two versions of the same thing that you want to install globally; there's no good way to do it. Ultimately this is because we've made the historical decision in operating systems and language environments that we shouldn't consider version numbers.

In Unix, there is only one thing that can be /usr/include/stdio.h, and only one thing that can be a particular major version of a shared library. In a language like Python, there can be only one thing that is what you get when you do 'import package'. If two Python programs are working in the same environment, they can't do 'import package' and get different versions of the module. This versionless view of various sorts of namespaces (header files, shared libraries, Python modules, etc) is convenient and humane (no one wants to do 'import package(version=....)'), but it makes it hard to support multiple versions.

The state of the art to support multiple "global" versions of the same thing is messy and complex, and as a result isn't widely used. With no system support for this sort of thing, language package managers have done the natural thing and rolled their own approaches to having different environments for different projects so they can have different versions of dependencies. For example, Python uses virtual environments, while Rust and Go gather dependencies in their build systems and statically link programs by default. And to be clear here, modern languages don't do this to be obstinate, they do it because attempting to have a single global environment has been repeatedly recognized as a mistake in practice (just look at Go's trajectory here for one painful example).

(At the operating system level, often people punt and use containers.)

To have a chance of unifying program managers and module managers, we would have to come up with an appealing, usable, humane solution to this problem. This solution somehow has to work well with existing systems, practices, and languages, rather than assuming changes to practices and language implementations, since such changes are unlikely (as always, social problems matter). At the very least, this seems like a tall order.

tech/PackageManagersTwoTypesII written at 23:39:37; Add Comment


Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.