I have recently been exploring some aspects of Python's virtual environments, so I thought I had a reasonable understanding of how they worked. In the HN discussion of my entry on installing modules to a custom location, I saw someone say that you could just run the virtual environment's python binary (ie, <directory>/bin/python, which is normally a symlink to the system Python) without 'activating' the virtual environment and it would still find the modules that you'd installed in the venv. This surprised me, because I had expected that you'd have to set some sort of environment variable to get Python to add your arbitrary non-standard location to sys.path. However, testing it showed that it really works, which is both convenient and magical.

(For me as a sysadmin, the important thing is that we can run programs that use a virtual environment without having to set any environment variables before hand. As long as we use the venv's Python, everything works, presumably even for things run from Unix cron or started as daemons. And we can arrange to always do that by setting '#!...' paths appropriately.)

At first I guessed that this was something that Python did in general if it found an appropriate directory tree around where you were running the python executable from. However, it turns out not to be quite this general. Although Python has a quite intricate process on Unix for finding its standard library and site-packages (see the comment at the start of Modules/getpath.c), it doesn't go so far as to add random trees just because they sort of look right. Instead, there is a special feature for Python virtual environments that looks for pyvenv.cfg and uses it to trigger some additional things. To quote from the source code comment:

Search for an "pyvenv.cfg" environment configuration file, first in the executable's directory and then in the parent directory. If found, open it for use when searching for prefixes.

(I haven't attempted to trace through the tangled code to determine exactly how this results in your venv's site-packages getting added to sys.path.)

Venv normally writes pyvenv.cfg to the root of your virtual environment directory tree (ie, in the parent of the executable's directory). For me the contents appear to be pretty generic; there's no mention of the virtual directory's location, and copying the pyvenv.cfg to the root of an artificially created minimal tree does cause its site-packages directory to get added to sys.path.

Since all of this is undocumented, it's probably best to consider it special private stuff that's only for the use of the standard venv system. If you want something like a virtual environment, complete with its own site-packages that will be automatically picked up when you run the Python from that directory hierarchy, just create a real virtual environment. They're pretty lightweight.

(Well, for a value of 'pretty lightweight' that amounts to 947 files, directories, and symbolic links and 12 Mbytes of disk space. Almost all of these are from installing pip into your new virtual environment; it drops to 13 files if you use --without-pip.)