Wandering Thoughts archives


Pipx and a problem with changing the system Python version

I use pipx on my work laptop, among other places, which I upgraded from Fedora 34 to Fedora 35 today. Afterward, my single pipx installed program didn't work, which was basically what I expected due to the familiar pip issue with Python versions; Fedora 34 has Python 3.9, while Fedora 35 has Python 3.10. Since virtual environments for one don't work with the other, the virtual environment for my installed programs couldn't find any Python packages that had been installed in them.

Since I've had success with 'pipx reinstall' before, I assumed that the way to fix this was to do a reinstall. Unfortunately this resulted in a spectacular failure, where pipx deleted my virtual environment then failed to recreate it with an error about pip not being available. Since the initial deletion lost the pipx metadata for my installed program there was no easy recovery, and anyway a 'pipx install' also had the 'pip not available' problem. Ultimately, this appears to be because pipx has a more or less hidden virtual environment of its own in ~/.local/pipx/shared, where it puts shared things that crucially includes pip itself. This virtual environment is also bound to a specific version of Python; if you change your Python, it too stops working, which means that any per-program virtual environments that point to it also stop working.

(You can find the signs of this in your venvs as a pipx_shared.pth file in each venv's lib/python3.X/site-packages/ directory, which has the absolute path to the relevant part of this shared venv. Note that this means that your pipx installed venvs will probably fail if you change your home directory or copy them to another system with a different home directory, because they have the absolute path to this shared tree.)

On my laptop, I fixed the problem by the brute force solution of removing ~/.local/pipx entirely, but I only had one program installed through pipx. I did experiment enough to determine that pipx will recreate ~/.local/pipx/shared if you delete it (or rename it), but I don't know if this will work through a complete installed Python version upgrade process. If it does, I think what you need to do is upgrade the Python you're using, delete ~/.local/pipx/shared, then do 'pipx reinstall-all'.

This is clearly a pipx bug where it should automatically detect an out of date shared area and rebuild it, but we deal with the pipx we have now, not the pipx we would like to have.

(This elaborates on some tweets.)

PS: Although pipx doesn't expose this to you, you can get a Python shell in the virtual environment of any installed program by running ~/.local/pipx/venv/<what>/bin/python. This may be useful if you want to do things like inspect that Python's sys.path setting.

python/PipxPythonVersionIssue written at 22:12:49; Add Comment

Page tools: See As Normal.
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.