The problem with trying to make everything into a Python module

March 22, 2013

One of the reasons for Django's unpleasant project restructuring is that they want your website directory (ie the directory that your project sits in) to be a module that can be imported. This in fact seems to be somewhat of a general trend; all sorts of things rather want you to to have not just a collection of files in a directory but an actual module. I wish they'd stop. Modules are not the be all and end all in Python, at least not as currently implemented, and not everything needs or wants to be a module.

The general reason for making things into modules is namespaces for imports. If you're sitting in your project's directory and do 'import fred', in theory this is ambiguous; you might mean your fred.py or you might mean some global fred module installed in Python. The absolute form of 'import mystuff.fred' is more or less unambiguous.

(This preference for modules also goes with the fact that the relative import syntax, 'from . import fred', is only valid in an actual module. I think that this is a terrible mistake, but no one asked me for my opinion.)

I have no problem with modules as such. The problem I have is how you get a directory to be a module, namely that you add the directory's parent to the Python search path (in one of a number of ways), and then the directory becomes a module (or technically I think a package) called its directory name. This is bad in at least two ways. It tightly couples together the directory name and the module name and it also makes everything else in the directory's parent available as a potential module. What both of these have in common is undesired name collisions. For example, you cannot be working on two versions of a 'fred' module that are sitting in a directory as, say, src/fred-1 and src/fred-2, not unless you want to have a src/fred symlink that you keep changing back and forth.

(The natural structure seems to be to isolate each module in its own artificial parent directory (eg src/fred-1/fred) or to ignore the whole issue, put everything in src/, and assume you will never have any collisions or be developing a new version of fred that you don't want src/bob getting when it does an 'import fred'.)

What would make this situation okay is a simple way to tell Python 'directory X is module Y', where 'X' might be '.' (the current directory). This should be available both on the Python command line and from inside Python code. Sadly I don't expect this to arrive any time soon.

(This stuff irritates me for reasons that are hard to pin down. Partly it just feels wrong (eg '/src' or wherever isn't a directory of modules, so why am I telling Python that it is?).)


Comments on this page:

From 87.79.78.105 at 2013-03-23 09:21:41:

By the custom in Perl you would have src/fred-1/lib/fred and src/fred-2/lib/fred.

This solves the feels-wrong issue too, because lib is a (sub)directory of modules, so you would be perfectly justified to tell Python that it is.

This also means you have a natural place for other things (e.g. src/fred-1/bin) which you may want to include with the package.

(Though I admit I wonder why you do have fred-1 and fred-2 rather than branches in your VCS.)

Aristotle Pagaltzis

From 108.44.40.236 at 2013-04-06 18:51:50:

You should be using 'virtualenv' to manage multiple python projects.

'virtualenvwrapper' makes things even easier.

   mkvirtualenv fred-1
   cd ~/src/fred-1
   python setup.py develop

   mkvirtualenv fred-2
   cd ~/src/fred-2
   python setup.py develop

...later..

  workon fred-1

or

  workon fred-2

- Justin

Written on 22 March 2013.
« The FreeBSD iSCSI initiator is not ready for serious use (as of 9.1)
Looking at how many recipients our average inbound email has »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Mar 22 00:18:26 2013
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.