Wandering Thoughts archives

Why people keep creating new package managers

Matt Simmons recently wrote Just what we need ... another package manager, in which he has an unhappy reaction to yet another language introducing yet another package manager. As a sysadmin I've long agreed with him for all sorts of reasons. Packaging and managing language 'packages' is an ongoing problem and in our environment it also causes user heartburn when we have to turn down requests to install language packages through the language's own mechanisms.

(We have a strong policy that we only install official distribution packages in order to keep our own sanity. This works for us but not necessarily for other people.)

But at the same time I have a lot of sympathy for the language people. Let's look at the problem from their perspective. Languages need:

• package management everywhere they run, possibly including Windows (which has no native package management system) and almost certainly including the Macs that many developers will be using (which also lack a native packaging system).

• something which doesn't force package contributors to learn more than one packaging system, because most people won't and languages want a healthy, thriving ecology of public packages. Ideally the one packaging system will be a simple, lightweight, and low friction one in order to encourage people to make and publish packages.

• for developers of language packages to not have to deal with the goat rodeos and political minefields that are the various distribution packaging processes, because making them do so is a great way of losing developers and not having packages.

• some relatively handy way to install and update packages that are not in the official distribution repositories. No language can really tolerate having its package availability being held hostage to the whims of distributions because it basically guarantees an out of date and limited package selection.

  (The interests of languages, developers of language packages, and distributions are basically all at odds here.)

• support for installing packages in non-default, non-system locations, ideally on both a 'per developer' and a 'per encapsulated environment' basis.

From the language's perspective it would be nice if package management for the language could optionally be done the same way regardless of what host you're on. In other words, developers should be able to use the same commands to install and set up packages on their development Macs as they do on testing VMs running some Linux distribution (or even FreeBSD), and possibly also on the production systems.

(In the modern lightweight world many small companies will not have actual sysadmins and developers will be setting up the production machines too for a while. Sysadmins do not like this but it is a reality. And languages are not designed for or used by sysadmins, they are mostly designed for developers, so it is not surprising that they are driven by the needs of developers.)

It's theoretically possible for a language's package system to meet all of these needs while still enabling distribution packages and doing as much as possible with the core distribution packaging system, either explicitly or behind convenient cross-platform cover scripts. However there are plenty of cases (like non-system installation of packages) that are simply not handled by the distribution packaging system and beyond that there are significant difficulties on both the technical and political levels. It is simply much easier for a language to roll its own packaging system.

(Ideally it will support creating distribution packages and using the distribution packaging mechanisms as well. Don't hold your breath; it's not going to be a language priority.)

The DNS TTL problem

It all started with a tweet by @Twirrim:

DNS TTL records exist for a reason. For the love of all that is holy, honour them. Don't presume to think you know better.

On the one hand, as a sysadmin I'm in full agreement with this view. I certainly want all of the DNS caches and recursive DNS servers out there to respect the TTLs we set on our DNS entries and it makes me irritated when people don't. On the other hand I also have to sympathize with the operators of DNS caches out there, because I rather suspect that there are a huge number of mis-set TTLs in practice.

The problem with DNS TTLs is that they are almost always an example of information that doesn't have to be correct, and we all know what eventually happens to such information. Most people's DNS entries change very rarely and are not looked up in any huge volume, so it doesn't really matter what TTLs they have. If they have the minimum TTL you won't notice the extra lookup volume and if they have an absurdly long TTL you won't notice the lingering old entries because you aren't changing your DNS entries anyways.

(And I'm not throwing stones here. We have a number of DNS entries with short TTLs that haven't changed for years in our zones, more or less just because. It would take work to go back through our zones, find them all, verify that we really don't need short TTLs any more, and take them out. It's simpler to let them sit there and it doesn't do us any harm.)

But I bet that operators of large scale DNS caches notice those things. I rather suspect that they get customer complaints when someone updates their DNS except that they had really long TTLs on the old entries and now the customers can't get to the new servers because the old entries are stilled cached. And I suspect that they notice the extra load from short TTLs forcing useful DNS entries to be discarded even when said DNS entries haven't actually been changed in the past year. I also suspect that there are more people doing DNS TTLs somewhat wrong than there are people doing them completely right. So I can see the engineering logic in overriding DNS TTLs in your large scale cache, however inconvenient it is for me as a sysadmin.

I don't have any answers to this and in a sense there are no answers. By that I mean that the large scale DNS caches that are currently monkeying around with people's DNS TTLs are not going to change their behavior any time soon, so the most I can do is live with it.

(Then there is the thornier issue of DNS lookups being remembered by long running programs that may have no idea of TTLs at all; instead they did a getaddrinfo() once and have held on to the result ever since. I suspect that web browsers no longer fall into this category, although they once did.)