Pipx and a problem with changing the system Python version

January 17, 2022

I use pipx on my work laptop, among other places, which I upgraded from Fedora 34 to Fedora 35 today. Afterward, my single pipx installed program didn't work, which was basically what I expected due to the familiar pip issue with Python versions; Fedora 34 has Python 3.9, while Fedora 35 has Python 3.10. Since virtual environments for one don't work with the other, the virtual environment for my installed programs couldn't find any Python packages that had been installed in them.

Since I've had success with 'pipx reinstall' before, I assumed that the way to fix this was to do a reinstall. Unfortunately this resulted in a spectacular failure, where pipx deleted my virtual environment then failed to recreate it with an error about pip not being available. Since the initial deletion lost the pipx metadata for my installed program there was no easy recovery, and anyway a 'pipx install' also had the 'pip not available' problem. Ultimately, this appears to be because pipx has a more or less hidden virtual environment of its own in ~/.local/pipx/shared, where it puts shared things that crucially includes pip itself. This virtual environment is also bound to a specific version of Python; if you change your Python, it too stops working, which means that any per-program virtual environments that point to it also stop working.

(You can find the signs of this in your venvs as a pipx_shared.pth file in each venv's lib/python3.X/site-packages/ directory, which has the absolute path to the relevant part of this shared venv. Note that this means that your pipx installed venvs will probably fail if you change your home directory or copy them to another system with a different home directory, because they have the absolute path to this shared tree.)

On my laptop, I fixed the problem by the brute force solution of removing ~/.local/pipx entirely, but I only had one program installed through pipx. I did experiment enough to determine that pipx will recreate ~/.local/pipx/shared if you delete it (or rename it), but I don't know if this will work through a complete installed Python version upgrade process. If it does, I think what you need to do is upgrade the Python you're using, delete ~/.local/pipx/shared, then do 'pipx reinstall-all'.

This is clearly a pipx bug where it should automatically detect an out of date shared area and rebuild it, but we deal with the pipx we have now, not the pipx we would like to have.

(This elaborates on some tweets.)

PS: Although pipx doesn't expose this to you, you can get a Python shell in the virtual environment of any installed program by running ~/.local/pipx/venv/<what>/bin/python. This may be useful if you want to do things like inspect that Python's sys.path setting.

Comments on this page:

By nyanpasu64 at 2022-01-18 10:27:18:

Yep, still broken. Someone reported it at https://github.com/pypa/pipx/issues/294, and I've commented my thoughts on it. It's frustrating that pipx doesn't fix broken shared or app envs to match the global Python version, and even worse that in some circumstances, pipx reinstall/reinstall-all will delete environments and fail to install them, then afterwards pipx acts like they were never installed. I hope it'll get fixed, but the maintainers don't seem to notice the issue and the users being affected. Maybe someday I'll write a fix myself... another item on my backlog I guess.

Written on 17 January 2022.
« HTTPS is still optional, at least sort of
Logs are invisible (at least most of the time and by default) »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Jan 17 22:12:49 2022
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.