== Into the depths of a Python _exec_ scope-handling bug I can rarely resist a Python mystery, so when I saw [[The exec Statement and A Python Mystery http://late.am/post/2012/04/30/the-exec-statement-and-a-python-mystery]] (via [[Hacker News http://news.ycombinator.com/item?id=3912390]]) my eyes lit right up. Here is the code in slightly shortened form (from [[here https://gist.github.com/2562350]], via [[HN http://news.ycombinator.com/item?id=3911238]] again): > def exec_and_return(codeobj, x): > exec codeobj > return x > > co1 = compile("x = x + 1", '', 'exec') > co2 = compile("del x", '', 'exec') > > print exec_and_return(co1, 1) > print exec_and_return(co2, 1) I'll save you running this in the interpreter and tell you the answer; this prints _2_ (what you'd expect) and then _1_ (not what you'd expect, you'd expect an error since _x_ is being referenced after having been deleted). The deep answer to what is going on here is that CPython's _exec_ imperfectly handles running code in a function scope, or specifically in matching up the scope between function scope and the compiled code's scope. The difficulties are ultimately caused by how CPython implements function scope (and accesses a function's local variables) at a low level. Used together this way, _compile()_ and _exec_ are specified to take code, compile it generically (without you having to specify what sort of scope it will run in), and then have _exec_ run that generically compiled code in the current scope, whatever that is. _exec_ can be called on to run the same compiled code object at the module ('global') level, in a class being created, or inside a function or a closure, and it must make all of them work. In most language implementations this would be straightforward because all of these different sorts of scopes would be implemented the same way (with the same internal data structures and the same way of accessing variables). With a uniform implementation of scopes, _exec_ and the compiled code wouldn't really care just what sort of scope it was running in; everything would work the same way regardless. Unfortunately for us, CPython doesn't work this way. In CPython, function local variables are stored and accessed in [[a completely different and much faster way WhyLocalVarsAreFast]] than variables are in all other scopes. Everything except functions holds variables in a dictionary (more or less) and accesses them by name; functions store local variables and function arguments in an indexed array and accesses them by their index number. This difference is reflected in the compiled code, where code in a function uses completely different bytecode operations to access variables than all other code does. This means that code compiled for a non-function context can't directly use a function's scope and vice versa (and in fact code compiled for one function can't use another functions's scope, because even if they have variable names in common the array offsets are probably going to be different). (Closures can use [[yet another set of bytecodes CPythonCellsClosures]] with somewhat peculiar effects.) Because it basically has no choice, _compile()_ creates code objects that are natively designed to run in a non-function scope; they access and manipulate variables through the name lookup bytecodes (you can see this by doing '_``import dis; dis.dis(co1.co_code)''_'). This means that when _exec_ runs a compiled code object in a function context, it must somehow fix up the difference between the array-based function scope that actually exists and the name-based scope that the compiled code expects. So what _exec_ does is that it materializes a fake name-based local scope (which is very similar to what you get from _locals()_), feeds it to the compiled code object as the local scope, and when the compiled code object finishes it copies all of the changes from the fake local scope object back to the real function scope. Except, of course, _exec_ has a bug; it doesn't copy *all* of the changes, as we've seen here. _exec_ will copy back changes in variable values (well, [[their bindings WhatVariablesMean]]) but it doesn't notice when variables have been deleted. So _co1_ changes the value of its local _x_ to 2 and this change is copied back to the function scope, making the function return 2, but when _co2_ deletes _x_ this deletion is not copied back; the function scope _x_ is left untouched at 1 and is then returned. (This bug is [[not intrinsic FLocalsAndTraceFunctions]] to how CPython allows function local variables to be manipulated, but fixing it would probably be complex.) === Sidebar: why _compile()_ has no choice The short answer is that using bytecode that's designed to run in a non-function scope is the only way that _compile()_ can create code objects that can at least sometimes run without fixups. Even if _compile()_ used the bytecodes for accessing a function's local variables, it doesn't know the proper array offsets for the various names; in fact, the proper array offsets will vary between functions. Consider: > def exec2(a, b, co): > x = a + b > exec co > return x In our original function, _x_ is in the second local variable slot. In _exec2_, _x_ is in the fourth local variable slot. Executing _co_ would thus need some sort of indirect mapping table for the slot offsets in order for it to manipulate the right variable. And of course _exec_ would have to make up a fake local scope and array if _co_ was run in a module context. Worse, the fixups needed in a module context would be quite challenging because the compiled code run by _exec_ can itself make calls to outside functions. When the code called out to any outside function, _exec_ would have to quickly copy any changes from the fake local array back to the modules's scope, call the outside function, then copy changes from the module's scope back into the fake local scope array. Let's just say no to that idea.