Why people keep creating new package managers

March 26, 2014

Matt Simmons recently wrote Just what we need ... another package manager, in which he has an unhappy reaction to yet another language introducing yet another package manager. As a sysadmin I've long agreed with him for all sorts of reasons. Packaging and managing language 'packages' is an ongoing problem and in our environment it also causes user heartburn when we have to turn down requests to install language packages through the language's own mechanisms.

(We have a strong policy that we only install official distribution packages in order to keep our own sanity. This works for us but not necessarily for other people.)

But at the same time I have a lot of sympathy for the language people. Let's look at the problem from their perspective. Languages need:

  • package management everywhere they run, possibly including Windows (which has no native package management system) and almost certainly including the Macs that many developers will be using (which also lack a native packaging system).

  • something which doesn't force package contributors to learn more than one packaging system, because most people won't and languages want a healthy, thriving ecology of public packages. Ideally the one packaging system will be a simple, lightweight, and low friction one in order to encourage people to make and publish packages.

  • for developers of language packages to not have to deal with the goat rodeos and political minefields that are the various distribution packaging processes, because making them do so is a great way of losing developers and not having packages.

  • some relatively handy way to install and update packages that are not in the official distribution repositories. No language can really tolerate having its package availability being held hostage to the whims of distributions because it basically guarantees an out of date and limited package selection.

    (The interests of languages, developers of language packages, and distributions are basically all at odds here.)

  • support for installing packages in non-default, non-system locations, ideally on both a 'per developer' and a 'per encapsulated environment' basis.

From the language's perspective it would be nice if package management for the language could optionally be done the same way regardless of what host you're on. In other words, developers should be able to use the same commands to install and set up packages on their development Macs as they do on testing VMs running some Linux distribution (or even FreeBSD), and possibly also on the production systems.

(In the modern lightweight world many small companies will not have actual sysadmins and developers will be setting up the production machines too for a while. Sysadmins do not like this but it is a reality. And languages are not designed for or used by sysadmins, they are mostly designed for developers, so it is not surprising that they are driven by the needs of developers.)

It's theoretically possible for a language's package system to meet all of these needs while still enabling distribution packages and doing as much as possible with the core distribution packaging system, either explicitly or behind convenient cross-platform cover scripts. However there are plenty of cases (like non-system installation of packages) that are simply not handled by the distribution packaging system and beyond that there are significant difficulties on both the technical and political levels. It is simply much easier for a language to roll its own packaging system.

(Ideally it will support creating distribution packages and using the distribution packaging mechanisms as well. Don't hold your breath; it's not going to be a language priority.)


Comments on this page:

By dozzie at 2014-03-27 05:01:05:

Add that some languages are actually hostile to any OS maintainers trying to package anything, like Ruby with its gems and RVM.

"(The interests of languages, developers of language packages, and distributions are basically all at odds here.)"

But what about the interests of the users and customers? They need (not necessarily in priority order)

1. Security fixes 2, Bug fixes 3. New features 4. A mechanism for having a voice in product direction.

I run a Fedora Linux desktop, but not every Perl or Python or Ruby or R package that I want to use is packaged as an RPM. This causes versioning nightmares for me, the user. I doubt if it's significantly better for Ubuntu, openSUSE or any other distro.

From my point of view as a developer, that last bullet point cannot be overstated.

For instance RedHat’s name in the Perl community is forever tarnished as the distribution that repeatedly shipped significantly broken /usr/bin/perls due to their insistence on patch-backporting fixes and sometimes features from later versions of Perl, an enterprise at which they had… irregular success. Other vendors have stripped out modules or the documentation from their main perl package.

As a result, over time, a culture of distrust toward the vendor-provided perl has grown. The attitude is that if you want predictability, your install your own local language environment to develop with or to deploy to, whereas the vendor-provided perl is seen as an opaque part of the OS which is maintained to satisfy the needs of vendor-maintained OS packages that need perl – not for the user’s application development needs.

Now RedHat have been good for a long time, and no other vendor has screwed up really badly in more recent times. But the culture persists since people soon realised that it’s also helpful to be isolate different Perl applications from each other. (An upgrade to a dependency used in several of your apps may cause conflicts in one of them but not the other, and you will likely want to not be forced to upgrade them in lock-step – because the consequence of that is getting stuck on ancient versions since nobody dares to upgrade for fear of breaking some other team’s code.)

Also, Perl’s release schedule has picked up steam into a steady rhythm in the last three years – which not all OS vendors have started running along with quite yet. 5.8 still has a huge install base only approached by 5.10, though the latter seems to be starting to fall off. So if you want to use a recent perl, or sometimes even just a perl younger than the iPhone, you have to install one yourself.

And all of these experiences generalise directly to any other source-based language.

And in thinking about this further, the issue it really comes down to is the package source/authority:

The reason for the friction experienced by Jane Perl-Girl is that she will often want to use things from CPAN – which get written on any number of systems and are not necessarily optimised to work with whatever particular OS (and version). But they tend to be written by people who use modules directly from CPAN, which means that in aggregate, software on CPAN tends to work best with other software on CPAN.

Likewise the vendor’s interest is to patch that perl and the modules in it in whatever way is necessary to make that perl and those modules work for the other packages the vendor is maintaining.

So if you want to use modules directly from the CPAN, you’ll want to do this on a perl downloaded from CPAN, using other modules directly from CPAN; whereas if you want to use packages that require perl modules, you’ll want to install the module packages provided by your OS vendor.

What this means to me is that this is that this multitude of package managers is not just a question of the situation at hand giving rise to certain pragmatics, but fundamentally inevitable/necessary.

Written on 26 March 2014.
« The DNS TTL problem
How we wound up with a RFC 1918 IP address visible in our public DNS »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Wed Mar 26 23:37:25 2014
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.