Python 3 venvs don't normally really embed their own copy of Python (on Unix)

September 17, 2020

Python 3 has standard support for virtual environments. The documentation describes them in general as:

[...] a self-contained directory tree that contains a Python installation for a particular version of Python, plus a number of additional packages.

As a system administrator, a 'self-contained directory tree' that has a particular version of Python is a scary thing to read about, because it implies that the person responsible for keeping that version of Python up to date on security patches, bug fixes, and so on is me, not my Unix distribution. I don't want to have to keep up with Python in that way; I want to delegate it to the fine people at Debian, Canonical, Red Hat, or whoever.

(I also don't want to have to keep careful track of all of the virtual environments we might be running so that I can remember to hunt all of them down to update them when a new version of Python is released.)

Fortunately it turns out that the venv system doesn't actually do this (based on my testing on Fedora 31 with Python 3.7.9, and also a bit on Ubuntu 18.04). Venv does create a <dir>/bin/python for you, but under normal circumstances this is a symlink to whatever version of Python you ran the venv module with. On Linux, by default this will be the system installed version of Python, which means that a normal system package update of it will automatically update all your venvs too.

(As usual, currently running processes will not magically be upgraded; you'll need to restart them.)

This does however mean that you can shoot yourself in the foot by moving a venv around or by upgrading the system significantly enough. The directory tree created contains directories that include the minor version of Python, such as the site-packages directory (normally found as <dir>/lib/python3.<X>/site-packages). If you upgrade the system Python to a new minor version (perhaps by doing a Linux distribution version upgrade, or by replacing the server with a new server running a more current version), or you move the venv between systems with different Python minor versions, your venv probably won't work because it's looking in the wrong place.

(For instance, Ubuntu 18.04 LTS has Python 3.6.9, Fedora 31 has Python 3.7.9, and Ubuntu 20.04 LTS has Python 3.8.2. I deal with all three at work.)

You can easily fix this with 'python3 -m venv --upgrade <DIR>', but you have to remember that you need to do this. The good news is that whatever is trying to use the venv is probably going to fail immediately, so you'll know right away that you need it.

PS: One way to 'move' a venv between systems this way is to have an environment with multiple versions of a Linux (as we do), and to build venvs on NFS filesystems that are mounted everywhere.

Written on 17 September 2020.
« How I think I want to drop modern Python packages into a single program
Python virtual environments transparently add themselves to sys.path »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Thu Sep 17 23:28:26 2020
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.