Wandering Thoughts archives

Use virtual environments to install third-party Python programs from PyPI

I've historically not been a fan of Python's virtual environments. While they're easy to set up these days, they're relatively heavyweight things (taking up tens of megabytes), they contain embedded references to the Python version they were built with, and it felt like vast overkill for simply installing a program (such as the Python LSP server and the third party packages it required from PyPI. So I've historically used Pip's "user" install mode ('pip install --user'), which puts everything into your $HOME/.local directory tree. However, I now feel that this is a mistake (although an attractive one). Instead, you should create virtual environments for any third party commands you install from PyPI or elsewhere.

The problem is that Pip's "user" mode involves pretending that Pip is basically a Unix distribution's package manager that just happens to be operating on your $HOME/.local. This is an attractive illusion and it sort of works, but in practice you run into issues over time when you upgrade things, especially if you have more than one program installed. You'll experience some of these issues with virtual environments as well, but with single purpose virtual environments (one venv per program) and keeping track of what you installed, the ultimate brute force solution is to delete and recreate the particular virtual environment. The dependency versions are getting tangled? Delete and recreate. You've moved to a new distribution version of Python (perhaps you've upgraded from one Ubuntu LTS to another)? It sounds like a good time to delete and recreate, rather than dealing with version issues..

More broadly, it feels to me that the Python packaging world is moving strongly toward using virtual environments as the solution to everything. As a result, I don't expect fundamental tools like Pip to spend much development effort on improving management of "user" mode installs. If anything, I expect Pip's user install mode to either quietly decay over time or to get deprecated at some point.

Since I've only recently come around to this view (after actively investigating the situation around upgrading programs with pip), I have no opinions on any of the programs and systems that are designed to make this easier. Pipx was mentioned in a comment by Tom on yesterday's entry, so I'll probably look at that first.

(I do think there are some uses for a Pip "user" install of PyPI packages, but that's for another entry.)