Closures versus classes for capturing state

March 9, 2006

Possibly like a lot of other people, I have a 'thread pool' Python module for doing decoupled asynchronous queries of things. The basic interface looks like this:

wp = WorkPool(HOWMANY)
for w in what:
  wp.enqueue(wrap_up(w))
wp.waitDone()
res = wp.getanswers()

The important operation is .enqueue(<THING>), which queues up <THING> to be called in a worker thread; its return value is saved, to be disgorged later by .getanswers().

To keep things simple, <THING> is called with no arguments. But usually what I really want to do is to call the same function with a bunch of different arguments, for example calling is_in_sbl for a bunch of IP addresses. To make this work, I need to capture the argument (or arguments) and create a simple callable object that remembers them; in this example, that's done by wrap_up.

There's two ways of wrapping up this sort of state in Python: classes (okay, objects) and closures. In the class-based approach, wrap_up is actually a class; its __init__ method saves w, and its __call__ method uses the saved w to do the actual function call. By contrast, in a closure-based approach wrap_up is a plain function that creates and returns closures, like this:

def wrap_up(w):
  return lambda: real_func(w)

The class approach is the more obvious and natural one, because the state saving is explicit. But when I wrote two structurally identical programs, the first one using a class and the second one with closures, I found that I plain liked closures better.

For me, the drawback of the class based approach is that it's too verbose. The same wrap_up would be:

class wrap_up:
  def __init__(self, w):
    self.w = w
  def __call__(self):
    return real_func(self.w)

(and it gets longer if you have more than one argument.)

I have a visceral reaction to verbosity; in cases like this it feels like make-work. The retort is that closures are far more magic than classes for most people and it's better to avoid magic, even if the result is more verbose.

For now, my view is that programming should be pleasant, not annoying, and that the verbosity of the class-based approach to state capture annoys me. So closures it is.

(Perhaps the real answer is that WorkPool should have an .enqueue_call(func, *args, **kwargs) method, as well as the plain .enqueue() one. This strays towards the Humane Interfaces arguments, though, so I'm a bit nervous.)

Written on 09 March 2006.
« Thinking about project failures
Making things simple for busy webmasters »

Page tools: View Source.
Search:
Login: Password:

Last modified: Thu Mar 9 02:50:11 2006
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.