Wandering Thoughts archives

2021-11-28

The problem I have with Pip's dependency version handling

Python's Pip package manager has a system where main programs and packages can specify the general versions of dependencies that they want. When you install a program through pip (either directly into a virtual environment or with a convenient tool like pipx), pip resolves the general version specifications to specific versions of the packages and installs them too. Like many language package managers, pip follows what I'll call a maximal version selection algorithm; it chooses the highest currently available version of dependencies that satisfy all constraints. Unfortunately I have come to feel that this is a bad choice for at least programs, for two reasons. One of the reasons is general and one of them is specific to pip's current capabilities and tooling.

The general reason is that it makes the installed set of dependencies not automatically reproducible. If I install the Python LSP server today and you install it a week from now, we may well not wind up with the same total version of everything even if the Python LSP server project hasn't released a new version. All it takes is a direct or indirect dependency to release a new version that's compatible with the version restrictions in the intervening week. Your pip install will pick up that new version, following pip's maximal version selection.

This is theoretically great, since you're getting the latest and thus best versions of everything. It is not necessarily practically great, since as we've all experienced, sometimes the very latest versions of things are not in fact the best versions, or at least the best versions in the context you're using them. If nothing else, you're getting a different setup than I am, which may wind up with confusing differences in behavior.

(For instance, your Python LSP server environment might have a new useful capability that mine doesn't. You'll tell me 'just do <X>', and I'll say 'what?'.)

The specific reason is that once I have pip install my version of something, pip doesn't really seem to provide a good way to update it to the versions of everything I'd get if I reinstalled today. That way, it would at least be easy for me and you to get the same versions of everything in our installs of the Python LSP server, which would let us get rid of problems (or at least let me see your problems, if more recent package versions have new problems). Pip has some features to try to do this, but in practice they don't seem to work very well for me. I'm left to do manual inspection with 'pip list --outdated', manual upgrades of things with 'pip install --upgrade', and then use of 'pip check' afterward to make sure that I haven't screwed up and upgraded something too far.

Pip is not going to change its general approach of maximal version selection (I think only Go has been willing to go that far). But I hope that someday either pip or additional tools have a good way to bring existing installs up to what they would be if reinstalled today.

(Pipx has its 'reinstall' option, but that's a blunt hammer and I'm not sure it works in all cases. I suppose I should try it someday on my Python LSP server installation, which has various additional optional packages installed too.)

PipDependencyVersionProblem written at 23:34:38; Add Comment


Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.