A confession: I find third party modules awkward in Python

September 14, 2010

One of my weaknesses is that with rare exceptions, I pretty much never use third party Python modules even when they'd make my life easier. I will sometimes use third party modules that are single files, but multi-file modules are generally too annoying (especially if they require compiling something).

The problem with third party modules is that installing and managing them is usually too much work, especially if they are large. You basically have four choices:

  • hope that all of the operating systems that you want to use the module on have prebuilt packages (which are sufficiently modern) and you can persuade your system administrator to install it.

  • if you are your system administrator, you can install the modules by hand (or by various Python packaging methods) in the system module area. This leads to sysadmin heartache and leaves you exposed if you ever find yourself wanting to use your program on a machine where you don't have this power.

  • put the third party modules straight into your program's main source directory (presuming that you have one, and that your program is not a single Python file before you started depending on third-party modules).

  • some module install schemes support an alternate installation location. If you take this option, you generally have to manually fiddle with sys.path in your programs to get them to find these modules (either directly or by setting various environment variables in wrapper scripts).

All of these approaches have aspects that annoy me. The friction they add is just enough that when I ask myself 'do I really need this module or is it just convenient', the answer usually is that it is just a convenience and I can live without it.

I'm aware that this is a personal twitch, and that a lot of people use the various install options happily. Probably they're more productive than me, too, because they're open to using convenient third party modules when they make sense. I did say this was a personal weakness.

Sidebar: how I'd like third party module installs to work

The basic answer is that I want to be able to configure a place (the root of a directory hierarch) for them to go, once, and then to have this place automatically added to the Python search path for everything I run so that my code can just do 'import <whatever>' without me having to do anything else.

I want this to happen without the need for environment variables or special cover scripts in front of the Python interpreter or whatever, because any of those requirements impose their own frictions. (For example, special environment variables make it annoying to run things from cron or other contexts where you don't have all of your environment automatically initialized for you.)


Comments on this page:

From 152.2.240.82 at 2010-09-14 10:23:17:

How about virtualenv? You could define environments (or just a single "sysadmin" environment) for your scripts and use pip to manage the modules which would then live inside that environment. For cron if you call the virtual environment's python it gets all the trappings (e.g. modules) of that environment without a need to activate it.

/path/to/virtualenv/bin/python /path/to/script.py

http://pypi.python.org/pypi/virtualenv

http://www.doughellmann.com/projects/virtualenvwrapper/

http://www.b-list.org/weblog/2008/dec/15/pip/

--Stephen Ball : http://xyzzyb.com

From 80.2.144.138 at 2010-09-14 20:01:33:

How about just dropping a .pth file into your site-packages directory pointing at your central install location? No environment variables or wrappers, just one file per python install.

By cks at 2010-09-14 21:28:54:

The problem with .pth files in the system site-packages directory is that if I do that, I'm effectively installing things system wide outside of the OS's packaging system (with all of the problems that that implies).

From 129.96.120.11 at 2010-09-16 00:04:40:

I use a proxy module in my source dir so i can dump 3rd party libs elsewhere. The "proxy" module is ridiculously simple:

examplelib.py

import os as _os # hide imports from the module's public namespace
execfile( _os.path.join( _os.path.dirname( __file__ ), '../ThirdParty/ExampleLib/examplelib.py' ) )

I put this proxy module in my source dir, so my code only needs to import examplelib without having to worry about sys.path. Note this supports relative paths, so you can have all your libs in source control and not worry about what your checkout path is. I haven't thought about how this would extend to importing a package though.

--Mark

Written on 14 September 2010.
« Why you want a recursive-forwarding caching nameserver
An overview of the Debian and RPM source package formats »

Page tools: View Source, View Normal.
Search:
Login: Password:

Last modified: Tue Sep 14 01:38:28 2010
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.