I wish I could split up code more easily in Python

March 16, 2016

This really starts with some tweets:

This Python program has grown to almost 1500 lines. I think I need an intervention, or better data structures, or something.
I also wish it was easier and more convenient to split up a Python program across multiple source files (it's one way Go wins).

The best way to split up a big program is to genuinely modularize it. In other words, find separate pieces of functionality that can be cleanly extracted and turn them into Python modules, in separate files. There are still issues with your main program actually finding the modules, but this can be worked around (even though it is and remains annoying).

However, this assumes that you have a modular structure to start with, with things sensibly separated. If your program started off as a little 200 line thing and then grew step by step into a 1500 line monster (especially iteratively), you may not necessarily have this. That's where Python makes things a little bit awkward. Splitting things up into separate files fundamentally puts them in separate modules and thus separate namespaces; in order to do it, you need to be able to pull your code apart in this way. If your code isn't in this state already you have some degree of rewriting ahead of you, and in the mean time you have a 1500 line Python file.

(In theory you can do 'from modname import *'. In practice this is only faking a single namespace and the fakery can break down in various ways.)

Go may be less elegant here (and Go certainly makes it harder to have separate namespaces), but you can slice a big source file up into several separate ones while keeping them all co-mingled as one module, all using bits and pieces from each other. Sometimes this is more convenient and expedient, even if it may be uglier.

With that said, Python has excellent reasons to require every separate file to be a separate module. To summarize very quickly, it's tied to how you don't just load a file of Python source code, you run it (with things like function and class definitions actually being executable statements, and possibly other interesting things happening). This is a straightforward model that's quite appropriate for an interpreted language, but it imposes certain constraints.


Comments on this page:

By Jukka at 2016-03-16 07:57:20:

I agree. Yet design is one thing; manual labour is another.

I continue to dislike Python due to the refactoring pain that comes with traditional text editors. That is, I like to do my Python with lots of terminals and Emacs, occasionally dropping in with another smaller editor.

Maybe I should change my habits (or learn LISP). And, well, like supposedly everyone, I do use Eclipse for Java, but the traditional Unix setup is otherwise so efficient for Python.

So you see bad design almost always when you look at your code, but often you just let it pass.

Written on 16 March 2016.
« An additional small detail of how writes work on ZFS raidzN pools
How 'from module import ...' is not doing what you may expect »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Wed Mar 16 01:43:45 2016
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.