Some notes on upgrading programs with Python's pip

September 15, 2021

My primary use of Python's pip package manager is to install programs like the Python LSP server; I may install these into either a contained environment (a virtual environment or a PyPy one) or as a user package with 'pip install --user'. In either case, the day will come when there's a new version of the Python LSP server (or whatever) and I want to update to it. As I noted down back in my pip cheatsheet, the basic command I want here is 'pip install --upgrade <package>', possibly with '--user' as well. However, it turns out that there are some complexities and issues here, which ultimately come about because pip is not the same sort of package manager as Fedora's DNF or Debian's apt.

The conventional way a Unix package manager such as DNF operates is that when you ask it to upgrade things, it upgrades everything that has new versions available. Pip doesn't behave this way. By default, it only upgrades the package that you asked it to, and doesn't upgrades any dependencies unless what you currently have doesn't satisfy the requirements of the new version of the upgraded package. Over enough time, this will often give you significantly out of date dependencies where you're missing out on improvements and bug fixes they've made even if your main package is theoretically fine with the old versions you have installed.

(This generally also means that the versions you've wound up with don't match what you'd get if you deleted everything and reinstalled from scratch, assuming you kept track of what top level packages you installed.)

This can be controlled (to some degree) by the --upgrade-strategy option to 'pip install'. This can be used to switch pip to an "eager" upgrade strategy, where it also upgrades dependencies to the latest available version that satisfies the requirements. You can set this as a default through pip's configuration files. However, this eager upgrading process is not flawless (as pip error messages may occasionally point out); you may wind up with versions of dependencies that satisfy the program, but not other things you have sitting around (including as globally installed modules from your operating system). Despite this, I've set up a ~/.config/pip/pip.conf to make the eager mode my default, because it's more like how I want package management for programs to work; normally I want to be using the latest and best version of everything.

(Usefully, you can tell if your pip.conf is working by what 'pip install --help' reports as the default for --upgrade-strategy.)

A Unix package manager like apt (almost) always removes the older version of a package when you upgrade to a newer one, and in fact many Unix packages can't have multiple versions installed at once. Pip somewhat apes this, but under some circumstances you can apparently wind up with older versions still present on disk. This especially matters if you decide to uninstall a package, because 'pip uninstall' seems to only remove the most recent version (what 'pip list' will show you as the version). If pip has left multiple versions sitting around, you may need a number of 'pip uninstall' invocations to get the package gone from 'pip list'. Alternatively, you can go to the appropriate location (such as '~/.local/lib/python3.X/site-packages') and manually remove all of the directories.

Comments on this page:

By Tom at 2021-09-16 03:46:35:

If you are primarily installing applications, you may want to consider using pipx ( instead of pip directly. It is a wrapper around pip and virtualenv, and is designed for installing applications. It tracks the top-level applications you install (and only installs the scripts.from them, and not their dependencies) and allows different apps to have conflicting dependencies installed.

By trcm at 2021-09-16 07:28:06:

I would mention you can set this with :

$ python3 -m pip config --user set global.upgrade-strategy eager

And check it with :

$ python3 -m pip config list
Written on 15 September 2021.
« There are (at least) two types of package managers
Use virtual environments to install third-party Python programs from PyPI »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Wed Sep 15 22:42:33 2021
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.