An issue with Pip installed packages and Python versions (on Unix)
Suppose, not hypothetically, that you want to install pyls, a LSP server for
Python, so that you can use it with (for example) GNU Emacs'
lsp-mode. Pyls is probably not packaged for
your Unix (it's not for Fedora or Ubuntu), but you can install it
with Pip (since it's on PyPi), either as
sudo pip install' to install it system wide (which may conflict
with your package manager) or as '
pip install --user' to install
it just for you.
(If this is a shared Unix machine, you probably need to do the latter.)
Then you upgrade your Unix version (or it gets upgraded), for example
from Fedora 31 to Fedora 32. Suddenly the
pyls program doesn't
work any more and even more puzzlingly, '
pip list --user' doesn't
even list anything. It's as if your personal installation of pyls
was somehow wiped out by the upgrade.
What's going is that pip installs things under a path that is specific to the minor version of Python, and when the minor version changes in the upgrade, the new version of Python doesn't find your old packages because it's looking in a different place. Fedora 31 had Python 3.7, which expects to find your personal packages in ~/.local/lib/python3.7/site-packages, where pip put them for you. Fedora 32 has Python 3.8, which expects to find the same packages in ~/.local/lib/python3.8/site-packages, and ignores the versions in python3.7/site-packages.
(The same thing happens on Ubuntu, where 18.04 LTS has 3.6.9 and 20.04 LTS has 3.8.5.)
As far as I can see there is no good way out of this. The same thing
happens if you install things system wide with '
sudo pip install'
(and I hope you kept notes on what you installed through pip and
what was already put there by the system). I think that it also
happens if you put pyls into a venv because venvs normally use
the system Python and inherit this version
specific site-packages directory.
(There is a '
python3 -m venv --upgrade <dir>' venv command to
upgrade the version of Python in a venv, but looking at the code
suggests that it doesn't do anything to migrate installed packages
to the new version. I can't test this, though, so perhaps I'm missing
My personal solution was to just rename the ~/.local/lib/python3.7 directory to 'python3.8'. Pip seems happy with the result, as does pyls. The more correct approach is probably to restart from scratch and reinstall all packages and programs like pyls.
(This elaborates on a tweet of mine. At the time of the tweet I hadn't realized that this applies to basically all uses of pip to install things, not just 'pip --user'.)