A problem in Python's implementation of closures

August 7, 2006

Python's implementation of closures for inner functions has a well known problem: you can't mutate the binding of captured outer variables. In other words, the following code does not work:

def cfunc(a):
  def _if(b):
    a = a + b
    return a
  return _if

There is nothing in the semantics of Python that require this result. Unlike the case of writing to a global variable in a function, what the a variable refers to in the scope of _if is completely unambiguous at all times.

I could make excuses for CPython, but the problem is pretty much there deliberately; while there is a special bytecode instruction (LOAD_DEREF) to read the value of captured variables in a closure, there is no bytecode instruction to write to them. In the absence of the ability to do anything else, the interpreter does its standard thing and treats any variable that is stored to in the function as function local (barring a global declaration).

The careful phrasing I have had to use in the first paragraph shows the way around this problem: while you can't change the binding of captured outer variables in a closure, you can mutate their value directly if the type of their value allows this. The classical way to do this is to make the desired variables into arrays, and then mutate the array contents. So we would write cfunc as:

def cfunc(a):
  t = [a]
  def _if(b):
    t[0] = t[0] + b
    return t[0]
  return _if

This version does what we want it to, at the expense of a certain amount of ugliness.

(Credit where credit is due department: I think I first saw the array trick in the sample WSGI server code in its specification.)

Comments on this page:

By DanielMartin at 2006-08-09 13:01:17:

Note that if you allow closures to rebind their environment, you're pretty much committing to a variable-as-storage-location model instead of a variable-as-bound-value model. As you've mentioned before, this is not the python way to do things.

Note that another language with the variable-as-bound-value model, Java, also requires a similar workaround if you want to pass a closure (in Java, an anonymous class) a spot that it can mutate, and when people need to do this in Java they often use a one element array.

Now Java, being one of those statically typed heavily-into-bondage languages, won't even let you compile a chunk of code that does this:

 // In the middle of some other stuff
 int a = 5;
 Runnable r = new Runnable(){public void run(){
   a = a*2;

So you never get the surprise of stuff misbehaving at runtime, you just get frustration at the compiler. The key is that java will allow an anonymous inner class to only touch variables from the surrounding environment that have been declared final, meaning that they can't be rebound. Simply saying final in front of int a will produce the (natural) error that the a = statement is problematic, since a is now final. The eventual solution is something like:

 // In the middle of some other stuff
 final int a[] = new int[] {5};
 Runnable r = new Runnable(){public void run(){
   a[0] = a[0]*2;

Which looks remarkably like what you had to do in python.

By cks at 2006-08-09 23:40:13:

Note that if you allow closures to rebind their environment, you're pretty much committing to a variable-as-storage-location model instead of a variable-as-bound-value model.

I think I'm missing something, because I don't see how this follows. Mutating what 'a' is bound to strikes me as not conceptually different from mutating what 'a[0]' is bound to, and Python already allows non-local binding mutation (functions can change the bindings of global variables).

(The CPython interpreter allows the captured function variables to be mutated outside of the closure and the closure will pick up the new binding, so directly mutating the value inside the closure is certainly technically possible.)

Written on 07 August 2006.
« A fun little regular expression bug
Slashdot's tacit admission of failure »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Aug 7 23:24:51 2006
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.