A Python program can be outside of a virtual environment it uses

March 2, 2022

A while ago I wrote about installing modules to a custom location, and in that entry one reason I said for not doing this with a virtual environment was that I didn't want to put the program involved into a virtual environment just to use some Python modules. Recently I realized that you don't have to, because of how virtual environments add themselves to sys.path. As long as you run your program using the virtual environment's Python, it gets to use all the modules you installed in the venv. It doesn't matter where the program is and you don't have to move it from its current location, you just have to change what 'python' it uses.

The full extended version of this is that if you have your program set up to run using '#!/usr/bin/env python3', you can change what Python and thus what virtual environment you use simply by changing the $PATH that it uses. The downside of this is that you can accidentally use a different Python than you intended because your $PATH isn't set up the way you thought it was, although in many cases this will result in immediate and visible problems because some modules you expected aren't there.

(One way this might happen is if you run the program using the system Python because you're starting it with a default $PATH. One classical way this can happen is running things from crontab entries.)

Another possible use for this, especially in the $PATH based version, is assembling a new virtual environment with new, updated versions of the modules you use in order to test your existing program with them. You can also use this to switch module versions back and forth in live usage just by changing the $PATH your program runs with (or by repeatedly editing its #! line, but that's more work).

Realizing this makes me much more likely in the future to just use virtual environments for third party modules. The one remaining irritation is that the virtual environment is specific to the Python version, but there are various ways of dealing with that. This is one of the cases where I think we're going to want to use 'pip freeze' (in advance) and then exactly reproduce our previous install in a new virtual environment. Or maybe we can get 'python3 -m venv --upgrade <venv-dir>' to work, although I'm not going to hold my breath on that one.

(A quick test suggests that upgrading the virtual environment doesn't work, at least for going from the Ubuntu 18.04 LTS Python 3 to the Ubuntu 20.04 LTS Python 3. This is more or less what I expected, given what would be involved, so building a new virtual environment from scratch it is. I can't say I'm particularly happy with this limitation of virtual environments, especially given that we always have at least two versions of Python 3 around because we always have two versions of Ubuntu LTS in service.)

Comments on this page:

One improvement on just saving the output of pip freeze that I use is a constraints file. This lets you have a requirements.txt file with your actual dependencies, but then save the exact versions you know work in a separate file. To get started you do pip install -r requirements.txt followed by pip freeze >constraints.txt inside your virtual environment. Then, when you want to do a repeatable install of those dependencies, you run pip install -r requirements.txt -c constraints.txt. I find it makes it a lot easier to come back to a project and figure out what dependencies it is safe to update because you separate out the things you depend on directly from the things you depend on transitively.

Written on 02 March 2022.
« The problem of keeping track of hardlinks as you traverse a directory tree
Understanding a thing with ZFS on Linux, kernel versions, RPMs, and DKMS »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Wed Mar 2 21:25:28 2022
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.