Wandering Thoughts archives

Pipx's 'reinstall' command works right by reinstalling injected packages too

While pipx is a good way to manage virtual environments for third party Python programs installed through PyPy, it necessarily still has some of the general issues with upgrading programs with pip. With pipx, though, the theoretically obvious solution to this is that if 'pipx upgrade' isn't making you happy any more, you can use 'pipx reinstall' to redo everything from scratch. However, one of my questions about pipx reinstall has always been if it would deal with a virtual environment, such as one for the Python LSP server, that also contains additional packages injected later.

(In the case of the Python LSP server, there are third party plugins. As far as I know you can only install these by injecting them into the LSP server virtual environment after the fact.)

Today, in a spirit of experimentation, I did a 'pipx reinstall python-lsp-server', after saving the actual Pip package list. To my happy surprise, pipx also reinstalled the extra injected packages for the third party plugins I want, and the only difference in the package lists is unimportant (at some point toml apparently stopped being part of someone's requirements). This means that having pipx reinstall things is generally going to be a reliable way of recreating a program's virtual environment with whatever package dependencies (and their versions) you 'should' have.

(For a version of 'should' that is 'what a person doing this from scratch would get today'. If you want to give someone an exact clone of your known to work virtual environment for a program, you need something else.)

This makes me happy, for obvious reasons. I do wish that the help documentation for 'pipx reinstall' actually said this explicitly, because the current phrasing silently implies that it doesn't also reinstall additional injected packages.

(The reinstall documentation says that the package is installed with 'pipx install ...'. A 'pipx install' obviously only installs the original package; it doesn't magically add any additional injected packages. Pipx is clearly doing something more than a plain install here, which is good, but it would also be good if it was documented.)

Using pipx in a PyPy installation more or less just works

I've decided that going forward I pretty much want to manage any third party Python programs I install, like the Python LSP server and pxpx, through pipx (as an extension of managing them in virtual environments). I've also experimented with installing PyPy myself. One of the advantages of using my own PyPy on our Ubuntu servers is that our Ubuntu servers run various versions of Ubuntu LTS, and so I would normally run into the pip Python version issue if I relied on the system Python. All of this (plus how pipx isn't packaged for Ubuntu 18.04) got me to wondering if pipx worked in PyPy. The answer is yes.

As you'd expect, you use the PyPy install's version of pip to install pipx, which winds up in the PyPy bin/ directory. When you run it, it creates and operates on your .local/pipx/venvs hierarchy and (still) puts links to the programs from installed packages into your .local/bin, instead of putting them in PyPy's bin/ directory. The installed packages do use your PyPy, not the system (C)Python, as you'd sort of expect. Running the pipx installed programs just works.

In a quick test, installing a (C)Python based package with the PyPy pipx works with 'pipx install --python ...'. Pipx lists the resulting venv as using the system version of Python, and the symlinks and so on in the venv point in the right place. This sort of thing may be necessary if you have some things (such as mypy) that don't really work under PyPy. Without testing it, I suspect that you can use several copies of pipx on the same pipx venv tree, so you can manage your CPython pipx things with the CPython pipx and the PyPy pipx things with your PyPy pipx.

Although I haven't gone through the exercise yet, I expect that the process of upgrading my version of PyPy will be to install the new PyPy, install pipx in the new PyPy version, and then run 'pipx reinstall-all' using the new pipx (or perhaps I will have to selectively reinstall only the venvs that are using PyPy). It's possible that there's a better way, since pipx's reinstall-all feels a bit brute force. On the other hand, there are probably a lot of bits of the pipx-created virtual environment that would have to be shuffled around to do it any other way, and pipx mostly makes it simple at the command level.

(I haven't tested 'pipx reinstall' on my Python LSP server venv, where I inject some extra optional packages on top of the LSP server package and its install options. It would be ideal if pipx reinjects those extra packages, but I'm not sure if it captures enough information in its metadata to do that.)