One of Python's problems with packages

December 7, 2008

One of the problems with Python's current approach to CPAN-like packages is, to put it in a particularly blunt way, that it hasn't quite sunk in that not everyone has root.

(I am aware that Python doesn't have an approach to packages as such; it's all done by distutils et al, although distutils is part of the standard library. That's part of the overall problem.)

Oh, sure, you can use distutils to install packages in places besides the system locations. And then your Python programs won't automatically find them so that you can import them; if you want to change this, you have to do various things to your programs or your environment (or both). The net result is that personally installed Python packages are significantly less useful than system-level packages.

(Also, even if you have root level access, there is always the CPAN problem.)

That it is non-trivial to use canned packages is, I think, one reason that something like the CPAN culture does not seem to have really caught on in the Python community. My own experience was that I would rather write my own WSGI system (admittedly partly for the learning experience) than go through the hassle of trying to pull in the existing wsgiref module (in the days before it was part of the standard library).

(In theory you can just dump simple modules into your program's or package's directory and import them directly without having to do any extra work. In practice, I want my package directory to just contain my code, so that I can keep track of it all.)

As I hinted earlier, one of the core problems is that handling packages has been left to Python modules. A really useful package system really does require the cooperation and involvement of CPython itself in some way; without this, there's only so much that distutils or anything else can do. However, I have to admit that I don't really know what CPython should do to improve things. It has fairly good reasons for not automatically running user specifc code or searching user package directories every time it starts, and there are mechanisms for overriding this; it's just that they're inconvenient, and I think that the inconvenience is high enough that they create real friction.

Comments on this page:

From at 2008-12-07 04:19:23:

That can’t be the reason for Python’s lack of CPAN: CPAN has the exact same problem. I think the reason is one of culture – the success of CPAN is administrative, not technical. Most people who go to build a replica are programmers who look at it from the point of view of local package management. CPAN, in contrast, was created by sysadmins who established a distribution network; the package manager and even the package format were pure afterthoughts. The Zen of Comprehensive Archive Networks talks about this at some length, although it doesn’t identify this point explicitly.

Aristotle Pagaltzis

From at 2008-12-07 04:28:37:

Python 2.6 comes with user-dir addition to the default PYTHONPATH; look at

From at 2008-12-07 07:50:46:

One of the most useful way of working with Python packages is to use virtualenv. This allows you to install Python modules and packages into a local environment without affecting your system Python. You can blow the whole environment away or even use it as an isolated deployment environment. Highly useful...

Michael Foord

By cks at 2008-12-07 13:54:54:

The 2.6 change is great news (and shows that I need to keep up more with recent versions of Python, even if I'm probably not going to have a system with 2.6 on it for a year or so). Hopefully distutils will change the default personal installation location to it (if it hasn't already).

The Zen of CPAN article is interesting reading, especially with your view of it. My feelings about the CPAN approach are biased, which makes me wonder if the sysadmin cultural factors that led to CPAN's success are a creation of its time, that time being before operating system package management really got off the ground.

(It's certainly given me things to think about.)

Written on 07 December 2008.
« On line endings and honesty
How to help programmers (part 1): the os.listdir() problem »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun Dec 7 00:43:53 2008
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.