Wandering Thoughts archives

A Python program can be outside of a virtual environment it uses

A while ago I wrote about installing modules to a custom location, and in that entry one reason I said for not doing this with a virtual environment was that I didn't want to put the program involved into a virtual environment just to use some Python modules. Recently I realized that you don't have to, because of how virtual environments add themselves to sys.path. As long as you run your program using the virtual environment's Python, it gets to use all the modules you installed in the venv. It doesn't matter where the program is and you don't have to move it from its current location, you just have to change what 'python' it uses.

The full extended version of this is that if you have your program set up to run using '#!/usr/bin/env python3', you can change what Python and thus what virtual environment you use simply by changing the $PATH that it uses. The downside of this is that you can accidentally use a different Python than you intended because your $PATH isn't set up the way you thought it was, although in many cases this will result in immediate and visible problems because some modules you expected aren't there.

(One way this might happen is if you run the program using the system Python because you're starting it with a default $PATH. One classical way this can happen is running things from crontab entries.)

Another possible use for this, especially in the $PATH based version, is assembling a new virtual environment with new, updated versions of the modules you use in order to test your existing program with them. You can also use this to switch module versions back and forth in live usage just by changing the $PATH your program runs with (or by repeatedly editing its #! line, but that's more work).

Realizing this makes me much more likely in the future to just use virtual environments for third party modules. The one remaining irritation is that the virtual environment is specific to the Python version, but there are various ways of dealing with that. This is one of the cases where I think we're going to want to use 'pip freeze' (in advance) and then exactly reproduce our previous install in a new virtual environment. Or maybe we can get 'python3 -m venv --upgrade <venv-dir>' to work, although I'm not going to hold my breath on that one.

(A quick test suggests that upgrading the virtual environment doesn't work, at least for going from the Ubuntu 18.04 LTS Python 3 to the Ubuntu 20.04 LTS Python 3. This is more or less what I expected, given what would be involved, so building a new virtual environment from scratch it is. I can't say I'm particularly happy with this limitation of virtual environments, especially given that we always have at least two versions of Python 3 around because we always have two versions of Ubuntu LTS in service.)