Suppose, not hypothetically, that you want to install pyls, a LSP server for Python, so that you can use it with (for example) GNU Emacs' lsp-mode. Pyls is probably not packaged for your Unix (it's not for Fedora or Ubuntu), but you can install it with Pip (since it's on PyPi), either as 'sudo pip install' to install it system wide (which may conflict with your package manager) or as 'pip install --user' to install it just for you.

(If this is a shared Unix machine, you probably need to do the latter.)

Then you upgrade your Unix version (or it gets upgraded), for example from Fedora 31 to Fedora 32. Suddenly the pyls program doesn't work any more and even more puzzlingly, 'pip list --user' doesn't even list anything. It's as if your personal installation of pyls was somehow wiped out by the upgrade.

What's going is that pip installs things under a path that is specific to the minor version of Python, and when the minor version changes in the upgrade, the new version of Python doesn't find your old packages because it's looking in a different place. Fedora 31 had Python 3.7, which expects to find your personal packages in ~/.local/lib/python3.7/site-packages, where pip put them for you. Fedora 32 has Python 3.8, which expects to find the same packages in ~/.local/lib/python3.8/site-packages, and ignores the versions in python3.7/site-packages.

(The same thing happens on Ubuntu, where 18.04 LTS has 3.6.9 and 20.04 LTS has 3.8.5.)

As far as I can see there is no good way out of this. The same thing happens if you install things system wide with 'sudo pip install' (and I hope you kept notes on what you installed through pip and what was already put there by the system). I think that it also happens if you put pyls into a venv because venvs normally use the system Python and inherit this version specific site-packages directory.

(There is a 'python3 -m venv --upgrade <dir>' venv command to upgrade the version of Python in a venv, but looking at the code suggests that it doesn't do anything to migrate installed packages to the new version. I can't test this, though, so perhaps I'm missing something.)

My personal solution was to just rename the ~/.local/lib/python3.7 directory to 'python3.8'. Pip seems happy with the result, as does pyls. The more correct approach is probably to restart from scratch and reinstall all packages and programs like pyls.

(This elaborates on a tweet of mine. At the time of the tweet I hadn't realized that this applies to basically all uses of pip to install things, not just 'pip --user'.)