== Some notes on Python packaging stuff that wasn't obvious to me

A comment by [[Lars Kellogg-Stedman http://blog.oddbit.com/]] on
[[this entry of mine ../linux/SSDDiskCacheDesire]] wound up with
me wanting to try out [[his lvcache utility
https://github.com/larsks/lvcache]], which is a Python program
that's packaged with a _setup.py_. Great, I thought, I know how to
install these things.

Well, no, not any more. While I wasn't looking, Python packaging
systems have gotten [[absurdly complex and annoying
http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-everywhere/]] (and
yes, one of the problems is that there are more than one of them).
My attempts to install lvcache (either privately or eventually
system-wide in a sacrificial virtual machine) failed in various
ways. In the process they left me very frustrated because I had
very little understanding of what a modern Python setup does when.
Since I now have somewhat more understanding I'm going to write up
what I know.

Once upon a time there was just _site-packages_ with _.py_ files
and plain directories in it, and life was simple and good. If you
wanted to you could augment the standard _site-packages_ by setting
_$PYTHONPATH_; the additional directories would be searched for
_.py_ files and plain directories too. Modern Python has added some
wrinkles:

* _.pth_ files list additional paths that will be used for importing
  things from (generally relative to the directory you find them in).
  These additional import paths are visible in _sys.path_, so if you're
  not sure if a _.pth_ file is working you can start Python and check
  what _sys.path_ reports.

  _.pth_ files in standard locations are loaded automatically; this
  includes your personal 'user' directory (on Unix, generally
  _$HOME/.local/lib/pythonX.Y/site-packages_, ie what '_python
  setup.py install --user_' et al will use). However, _.pth_ files
  in directories that are merely on your _$PYTHONPATH_ are not
  automatically loaded by Python and must be bootstrapped somehow;
  if you use ((easy_install --prefix)), it will stick a _site.py_
  file to do this in the directory.

  (There are some really weird things that go on with _.pth_
  files. See [[Armin Ronacher
  http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-everywhere/]].)

* _.egg_ *files* are ZIP files, [[which Python can import code from
  directly RunningZipfiles]]. They contain metadata and a module
  directory with _.py_ files and normally appear directly on
  _sys.path_ (eg the _.egg_ file is listed itself).
  You can inspect _.egg_ file contents with '_unzip -v thing.egg_'.
  Under some circumstances it's possible for the install process to
  build a _.egg_ that doesn't contain any Python code (or contains
  incomplete Python code); if you're facing mysterious failures,
  you may need to check for this.

* _.egg_ directories are unpacked versions of the ZIP versions above.
  I don't know when ((easy_install)) et al create directories versus
  files. Like the files they appear on _sys.path_ directly. They can
  be inspected directly.

Modern installers no longer just put files and module directories
in places. Instead, they make or obtain eggs and install the eggs.
The good news is that things like ((easy_install)) follow dependencies
(assuming that everyone has properly specified them, not always a
given). The bad news is that this is much less inspectable than the
old days.

(Okay, the other good news is that you can see which version of
what you've installed by hand, instead of having a mess of stuff.)

In a properly functionally installed environment you should be able
to fire up an interactive Python session and do '_import <module>_'
for every theoretically installed module. If this fails, either any
_.pth_ files are not getting bootstrapped (which can be checked by
looking at _sys.path_), you don't have a module installed that you
think you should, or perhaps the module is empty or damaged.

I'm sure all of this is documented in one or more places in the
official Python documentation, but it is sure not easy to find
if it is (and I really don't think there's one place that puts
it all together).

PS: if you're installing a local copy of a package's source you
want '((easy_install .))' (in the source directory), likely with
_--user_ or _--prefix_.  At least some of the time, ((easy_install))
will insist that you precreate the _--prefix_ directory for it; it
will always insist that you add it to _$PYTHONPATH_.

(The current anarchy of Python packaging and install systems requires
another rant but I am too exhausted for it right now.)