Use virtual environments to install third-party Python programs from PyPI
I've historically not been a fan of Python's virtual environments. While they're easy
to set up these days, they're relatively heavyweight things (taking
up tens of megabytes), they contain embedded references to the
Python version they were built with, and it
felt like vast overkill for simply installing a program (such as
the Python LSP server and the third party
packages it required from PyPI. So I've
historically used Pip's "user" install mode ('
pip install --user'),
which puts everything into your
$HOME/.local directory tree.
However, I now feel that this is a mistake (although an attractive
one). Instead, you should create virtual environments for any third
party commands you install from PyPI or elsewhere.
The problem is that Pip's "user" mode involves pretending
that Pip is basically a Unix distribution's package manager that just happens to be operating
$HOME/.local. This is an attractive illusion and it sort
of works, but in practice you run into issues over time when you
upgrade things, especially if you have more
than one program installed. You'll experience some of these issues
with virtual environments as well, but with single purpose virtual
environments (one venv per program) and keeping track of what you
installed, the ultimate brute force solution is to
delete and recreate the particular virtual environment. The dependency
versions are getting tangled? Delete and recreate. You've moved to a new
distribution version of Python (perhaps you've upgraded from one Ubuntu
LTS to another)? It sounds like a good time to delete and recreate,
rather than dealing with version issues..
More broadly, it feels to me that the Python packaging world is moving strongly toward using virtual environments as the solution to everything. As a result, I don't expect fundamental tools like Pip to spend much development effort on improving management of "user" mode installs. If anything, I expect Pip's user install mode to either quietly decay over time or to get deprecated at some point.
Since I've only recently come around to this view (after actively investigating the situation around upgrading programs with pip), I have no opinions on any of the programs and systems that are designed to make this easier. Pipx was mentioned in a comment by Tom on yesterday's entry, so I'll probably look at that first.
(I do think there are some uses for a Pip "user" install of PyPI packages, but that's for another entry.)