Wandering Thoughts archives

Phase objects: simple decent error reporting for Python programs

For a lot of the sort of utility programs I write in Python, the only thing the program can really do when it runs into an operating system level error is report whatever the problem is and exit. So programs start out as one big try:/except: block:

try:
    fp1 = open(fnameA, "r")
    ....
    fp2 = open(fnameB, "r")
    ....
except EnvironmentError, e:
    die("OS error: %s" % str(e))

The problem with this is that the error reported, while accurate, is basically without context; you don't know what the program was doing when it reported 'permission denied' or whatever. So simple programs mutate slightly to keep track of what they are doing in a variable:

try:
    phase = "opening %s" % fnameA
    fp1 = open(fnameA, "r")
    phase = "parsing %s" % fnameA
    ....
except EnvironmentError, e:
    die("%s: error: %s" % (phase, str(e)))

(Well, my simple programs do, since I have no desire to wrap each separate operation that might fail in its own try:/except: block. That's too much work.)

The problem with this approach is that it doesn't work very well when your program grows subroutines, because updating a global variable to track the phase is a pain. The solution I have adopted is to create a special class to track what phase of execution the program is in; with my typical (lack of) imagination, I call this class Phase, and it looks like this:

class Phase(object):
    def __init__(self):
        self.phase = ["starting",]
    def __call__(self, ph):
        self.phase = [ph, ]
    def __str__(self):
        return ": ".join(self.phase)
    def push(self, ph):
        self.phase.append(ph)
    def pop(self):
        self.phase.pop()

It is used as phase("reading file whatever"). It supports push() and pop() operations so that you can have nested situations (such as the ability of configuration files to include other configuration files) that report the full chain of events of how you wound up doing whatever it was that failed.

My programs just make a single global Phase instance and then use it directly as a global variable, but I suppose a more elegant and reusable method would be to create the instance at the top level and pass it down through the call chain.

In praise of Python's Global Interpreter Lock

It's common to say bad things about the GIL (a round of GIL bashing happened recently, for example). I have a contrarian view: I think that the GIL is the right thing for Python.

Python needs some concurrency support, if only so that it can make inherently blocking operations into non-blocking ones. At the same time, the thread and shared memory based concurrent programming model is a terrible one that programmers should avoid as much as possible.

The GIL is good enough and simple enough to easily do the former (in fact it's the only thing you should use the GIL for), while being crappy enough to discourage programmers from writing actual thread and shared memory based concurrent programs, partly because their programs don't get much benefit.

Worse thread support would make the blocking to non-blocking transformation too painful. Better thread support would tempt Python programmers into more actual thread-based concurrent programming, with the attendant heartburn. The GIL sits in a sweet spot in the middle: just good enough to be useful yet pointless enough to not be used.

(In the process the GIL optimizes for the single-threaded case, which is likely to be the most common one for quite some time even in the theoretical glorious multi-core future.)

The GIL also means that the core Python developers are not forced to spend a lot of effort supporting a model of concurrency that will ultimately be a dead end. (If shared memory threading is not a dead end for most concurrency work, we are in a lot of trouble.)