Some early notes on using pipx for managing third-party Python programs

October 5, 2021

Recently I wrote about how I should use virtual environments for third party programs, and said that I would likely wind up trying pipx. Today, I ran 'pip3 list --user --outdated', stared at the output of that and 'pip3 list --user', and decided that my situation in my ~/.local had reached enough critical mass that I was willing to burn it down and start over from scratch. So here are some early notes on pipx, after experimenting with it on my laptop and then switching to it on my home desktop.

In general, pipx is pleasantly straightforward to use in a basic way. Running 'pipx install mypy' (for example) creates a .local/pipx/venvs/mypy virtual environment, installs things into it, and then creates symlinks in .local/bin to the various programs mypy normally installs. Pipx adds a 'pipx_metadata.json' file to the venv with various pieces of information about what the virtual environment contains. The venvs appear to be named after the main package you installed with 'pipx install'.

One of my major uses of pipx is to install the Python LSP server, which has two additional things. The first is that you normally install it with some additional pieces specified, not just as a plain package name, for example:

pipx install 'python-lsp-server[rope,yapf]'

Pipx coped with this package specification fine, and called the venv 'python-lsp-server'. The second thing is that there are additional third party plugins you may want to add, like mypy-ls; these need to be installed into the venv somehow. In pipx, this is done with 'pipx inject <package>', eg 'pipx inject mypy-ls'.

All of this appears to be recorded in pipx_metadata.json, so I can hope that 'pipx reinstall <...>' will correctly reinstall all of my Python LSP setup, without me having to remember exactly what I added (I haven't tested this yet, and the manual page is ambiguous).

I'm not sure how you're supposed to look for upgrades to the things you have installed. I suppose one option is 'pipx upgrade' or 'pipx upgrade-all' and just letting it do things, and another is to directly ask pip, using 'pipx runpip <package> list --outdated'. The last is probably the only way to find out about dependencies with new versions you may want, which pip doesn't normally upgrade anyway. One brute force way of dealing with the general issues of upgrading programs with pip is to just periodically run 'pipx reinstall <...>', which should give you a venv that is as up to date as possible at the expense of a certain amount of overhead.

Overall, even if pipx isn't perfect it's a lot better than the mess that was my old ~/.local and actually using it is sufficiently easy and non-annoying that I'm willing to put up with minor flaws if it has them.

Comments on this page:

I don't understand. If pipx is a wrapper for virtualenv, what are you getting out of using it instead of virtualenv directly? Now apart from pip's own commands, you will have to keep pipx's own commands in head.

If you use virtualenv-wrapper, you get mkvirtualenv and workon aliases which essentially lets you do what pipx does. Plus you now get to use regular pip for installation and pip freeze > requirements.txt for saving deps.

By Seth at 2021-10-06 10:52:48:

Since I have to reinstall pipx-installed tools with most OS updates (due to the python version), I just keep a little script full of "pipx install" commands to rebuild it all. It lives next to my "go install" script and "npm install" script, although those don't need to be run as often.

Sandip: virtualenv-wrapper looks more like something for easily working in different venvs -- dev work. pipx is a package manager for installing tools, where you wouldn't otherwise even know it's python. You just want to be able to run the command it provides.

By cks at 2021-10-06 12:44:31:

One quick note: 'pip freeze' is too specific for what I want, since it records all current packages, including dependencies, and it locks their versions.

Sandip, pipx is effectively one level above regular virtualenvwrapper, allowing you to end up with a nice single bin/ folder containing (symlinks to) all of your "incidentally written in Python" CLI tools, each of which is safely corralled into its own virtualenv. Thus they each may be upgraded/removed/reinstalled/etc without harming any of the others.

With straight venvwrapper you have to do all of that management yourself, including identifying exactly which binstubs a package has installed are the ones you want to end up in your global $PATH. (pipx does this by interpreting the package you specifically requested's setuptools metadata - so it won't end up polluting this bin/ dir with that tool's dependencies' binstubs, etc etc.)

Or you would keep One Big Gnarly Virtualenv for "your cli tools and stuff" -- which will quickly turn into a hot mess even in the best of times, to say nothing of when 2-3 of your CLI tools disagree about which 'requests', 'cryptography' or 'rich' to use...

Written on 05 October 2021.
« The "why" problem with on-host (host-based) firewalls on your machines
The OpenSSH server has limits on what user authentication you can use »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Tue Oct 5 23:59:15 2021
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.