Chris's Wiki :: blog/python/PipPythonVersionIssue

An issue with Pip installed packages and Python versions (on Unix)

October 28, 2020

Suppose, not hypothetically, that you want to install pyls, a LSP server for Python, so that you can use it with (for example) GNU Emacs' lsp-mode. Pyls is probably not packaged for your Unix (it's not for Fedora or Ubuntu), but you can install it with Pip (since it's on PyPi), either as 'sudo pip install' to install it system wide (which may conflict with your package manager) or as 'pip install --user' to install it just for you.

(If this is a shared Unix machine, you probably need to do the latter.)

Then you upgrade your Unix version (or it gets upgraded), for example from Fedora 31 to Fedora 32. Suddenly the pyls program doesn't work any more and even more puzzlingly, 'pip list --user' doesn't even list anything. It's as if your personal installation of pyls was somehow wiped out by the upgrade.

What's going is that pip installs things under a path that is specific to the minor version of Python, and when the minor version changes in the upgrade, the new version of Python doesn't find your old packages because it's looking in a different place. Fedora 31 had Python 3.7, which expects to find your personal packages in ~/.local/lib/python3.7/site-packages, where pip put them for you. Fedora 32 has Python 3.8, which expects to find the same packages in ~/.local/lib/python3.8/site-packages, and ignores the versions in python3.7/site-packages.

(The same thing happens on Ubuntu, where 18.04 LTS has 3.6.9 and 20.04 LTS has 3.8.5.)

As far as I can see there is no good way out of this. The same thing happens if you install things system wide with 'sudo pip install' (and I hope you kept notes on what you installed through pip and what was already put there by the system). I think that it also happens if you put pyls into a venv because venvs normally use the system Python and inherit this version specific site-packages directory.

(There is a 'python3 -m venv --upgrade <dir>' venv command to upgrade the version of Python in a venv, but looking at the code suggests that it doesn't do anything to migrate installed packages to the new version. I can't test this, though, so perhaps I'm missing something.)

My personal solution was to just rename the ~/.local/lib/python3.7 directory to 'python3.8'. Pip seems happy with the result, as does pyls. The more correct approach is probably to restart from scratch and reinstall all packages and programs like pyls.

(This elaborates on a tweet of mine. At the time of the tweet I hadn't realized that this applies to basically all uses of pip to install things, not just 'pip --user'.)

Comments on this page:

By rjc at 2020-10-28 03:39:09:

This is an issue we encounter time and time again and it is the very reason why the first thing we tell the students, apart from suggesting to use virtual environments, is to always be as explicit as possible and use:
   python3.8 -m pip install --user foo
It'll work on both different machines (we use NFS) and on newer OS releases (Ubuntu 16.04+ LTS) - we use Python from "deadsnakes" PPA so at there are at least 4 Python 3 releases available to choose from.

https://docs.python.org/3/installing/index.html#work-with-multiple-versions-of-python-installed-in-parallel

By Albert at 2020-10-28 05:49:29:

The few times it happened to me, I just re-did pip install in the new environment and delete the old copies.

By Leah Neukirchen at 2020-10-28 07:46:33:

pip list --user should list your packages, potentially combined with --not-required to get the manually installed packages only.

Renaming will only work if there are no binary libraries, updating a minor version of Python will require rebuilding those.

By cks at 2020-10-28 13:52:14:

The problem with pip list --user is that you need to know in advance that you need to record your package list. Once the upgrade has happened, it's too late; the new version of Python and pip don't see your old version's packages to list them, so you get to dig out the list by hand.

By rwoodsmall at 2020-10-28 17:29:35:

There are two options:

- use a virtualenv/venv in your home directory and risk it becoming out of date - compile your own Python, again in your home directory or some shared location

"System" Python is just that, and basically only fit for that purpose - think yum, bits of dnf, etc. From a developer perspective, you have to own your Python install, and that means either fully utilizing the package manager or not using it at all. There's not really a comfortable space in between.

By wilbur at 2020-10-29 12:08:54:

This has also annoyed me in the past! FWIW, setting the env var PYTHONUSERBASE will override the default value for `site.USER_BASE`:
$ PYTHONUSERBASE=/tmp/my-user-base python3 -m pip install --user black
...
$ ls /tmp/my-user-base/bin/
black        black-primer blackd

Written on 28 October 2020.

«	Sometimes alerts have inobvious reasons for existing

An illustration of why running code during import is a bad idea (and how it happens anyway) »