Some notes on Python packaging stuff that wasn't obvious to me
A comment by Lars Kellogg-Stedman on
this entry of mine wound up with
me wanting to try out his lvcache utility, which is a Python program
that's packaged with a
setup.py. Great, I thought, I know how to
install these things.
Well, no, not any more. While I wasn't looking, Python packaging systems have gotten absurdly complex and annoying (and yes, one of the problems is that there are more than one of them). My attempts to install lvcache (either privately or eventually system-wide in a sacrificial virtual machine) failed in various ways. In the process they left me very frustrated because I had very little understanding of what a modern Python setup does when. Since I now have somewhat more understanding I'm going to write up what I know.
Once upon a time there was just
and plain directories in it, and life was simple and good. If you
wanted to you could augment the standard
site-packages by setting
$PYTHONPATH; the additional directories would be searched for
.py files and plain directories too. Modern Python has added some
.pthfiles list additional paths that will be used for importing things from (generally relative to the directory you find them in). These additional import paths are visible in
sys.path, so if you're not sure if a
.pthfile is working you can start Python and check what
.pthfiles in standard locations are loaded automatically; this includes your personal 'user' directory (on Unix, generally
$HOME/.local/lib/pythonX.Y/site-packages, ie what '
python setup.py install --user' et al will use). However,
.pthfiles in directories that are merely on your
$PYTHONPATHare not automatically loaded by Python and must be bootstrapped somehow; if you use
easy_install --prefix, it will stick a
site.pyfile to do this in the directory.
(There are some really weird things that go on with
.pthfiles. See Armin Ronacher.)
.eggfiles are ZIP files, which Python can import code from directly. They contain metadata and a module directory with
.pyfiles and normally appear directly on
.eggfile is listed itself). You can inspect
.eggfile contents with '
unzip -v thing.egg'. Under some circumstances it's possible for the install process to build a
.eggthat doesn't contain any Python code (or contains incomplete Python code); if you're facing mysterious failures, you may need to check for this.
.eggdirectories are unpacked versions of the ZIP versions above. I don't know when
easy_installet al create directories versus files. Like the files they appear on
sys.pathdirectly. They can be inspected directly.
Modern installers no longer just put files and module directories
in places. Instead, they make or obtain eggs and install the eggs.
The good news is that things like
easy_install follow dependencies
(assuming that everyone has properly specified them, not always a
given). The bad news is that this is much less inspectable than the
(Okay, the other good news is that you can see which version of what you've installed by hand, instead of having a mess of stuff.)
In a properly functionally installed environment you should be able
to fire up an interactive Python session and do '
for every theoretically installed module. If this fails, either any
.pth files are not getting bootstrapped (which can be checked by
sys.path), you don't have a module installed that you
think you should, or perhaps the module is empty or damaged.
I'm sure all of this is documented in one or more places in the official Python documentation, but it is sure not easy to find if it is (and I really don't think there's one place that puts it all together).
PS: if you're installing a local copy of a package's source you
easy_install .' (in the source directory), likely with
--prefix. At least some of the time,
will insist that you precreate the
--prefix directory for it; it
will always insist that you add it to
(The current anarchy of Python packaging and install systems requires another rant but I am too exhausted for it right now.)