Wandering Thoughts archives

Into the depths of a Python exec scope-handling bug

I can rarely resist a Python mystery, so when I saw The exec Statement and A Python Mystery (via Hacker News) my eyes lit right up. Here is the code in slightly shortened form (from here, via HN again):

def exec_and_return(codeobj, x):
  exec codeobj
  return x

co1 = compile("x = x + 1", '<s>', 'exec')
co2 = compile("del x", '<s>', 'exec')

print exec_and_return(co1, 1)
print exec_and_return(co2, 1)

I'll save you running this in the interpreter and tell you the answer; this prints 2 (what you'd expect) and then 1 (not what you'd expect, you'd expect an error since x is being referenced after having been deleted).

The deep answer to what is going on here is that CPython's exec imperfectly handles running code in a function scope, or specifically in matching up the scope between function scope and the compiled code's scope. The difficulties are ultimately caused by how CPython implements function scope (and accesses a function's local variables) at a low level.

Used together this way, compile() and exec are specified to take code, compile it generically (without you having to specify what sort of scope it will run in), and then have exec run that generically compiled code in the current scope, whatever that is. exec can be called on to run the same compiled code object at the module ('global') level, in a class being created, or inside a function or a closure, and it must make all of them work. In most language implementations this would be straightforward because all of these different sorts of scopes would be implemented the same way (with the same internal data structures and the same way of accessing variables). With a uniform implementation of scopes, exec and the compiled code wouldn't really care just what sort of scope it was running in; everything would work the same way regardless. Unfortunately for us, CPython doesn't work this way.

In CPython, function local variables are stored and accessed in a completely different and much faster way than variables are in all other scopes. Everything except functions holds variables in a dictionary (more or less) and accesses them by name; functions store local variables and function arguments in an indexed array and accesses them by their index number. This difference is reflected in the compiled code, where code in a function uses completely different bytecode operations to access variables than all other code does. This means that code compiled for a non-function context can't directly use a function's scope and vice versa (and in fact code compiled for one function can't use another functions's scope, because even if they have variable names in common the array offsets are probably going to be different).

(Closures can use yet another set of bytecodes with somewhat peculiar effects.)

Because it basically has no choice, compile() creates code objects that are natively designed to run in a non-function scope; they access and manipulate variables through the name lookup bytecodes (you can see this by doing 'import dis; dis.dis(co1.co_code)'). This means that when exec runs a compiled code object in a function context, it must somehow fix up the difference between the array-based function scope that actually exists and the name-based scope that the compiled code expects. So what exec does is that it materializes a fake name-based local scope (which is very similar to what you get from locals()), feeds it to the compiled code object as the local scope, and when the compiled code object finishes it copies all of the changes from the fake local scope object back to the real function scope.

Except, of course, exec has a bug; it doesn't copy all of the changes, as we've seen here. exec will copy back changes in variable values (well, their bindings) but it doesn't notice when variables have been deleted. So co1 changes the value of its local x to 2 and this change is copied back to the function scope, making the function return 2, but when co2 deletes x this deletion is not copied back; the function scope x is left untouched at 1 and is then returned.

(This bug is not intrinsic to how CPython allows function local variables to be manipulated, but fixing it would probably be complex.)

Sidebar: why compile() has no choice

The short answer is that using bytecode that's designed to run in a non-function scope is the only way that compile() can create code objects that can at least sometimes run without fixups. Even if compile() used the bytecodes for accessing a function's local variables, it doesn't know the proper array offsets for the various names; in fact, the proper array offsets will vary between functions. Consider:

def exec2(a, b, co):
  x = a + b
  exec co
  return x

In our original function, x is in the second local variable slot. In exec2, x is in the fourth local variable slot. Executing co would thus need some sort of indirect mapping table for the slot offsets in order for it to manipulate the right variable. And of course exec would have to make up a fake local scope and array if co was run in a module context.

Worse, the fixups needed in a module context would be quite challenging because the compiled code run by exec can itself make calls to outside functions. When the code called out to any outside function, exec would have to quickly copy any changes from the fake local array back to the modules's scope, call the outside function, then copy changes from the module's scope back into the fake local scope array. Let's just say no to that idea.