Wandering Thoughts archives

Why I wound up using Django

Eevee just wrote (and I just read) a good article on Python web development. It was an interesting experience for me, because a great deal of the article is about a middle level in Python webdev that I've basically skipped right over. I've worked at the level of CGI and WSGI and I've worked in the big framework land of Django, but I've never played around in the world of relatively simple and component-ized Python web frameworks that Eevee mostly talks about (to the extent that I merged them in with Django when I wrote about modern Python web app options). And that raises an obvious question: why did I wind up using Django instead of one of the smaller and simpler alternatives?

While I could say various things, the honest answer is twofold: good PR on Django's part and 'batteries included', with the first one being more important. The good PR insured that I knew of Django and indeed thought of it as the natural choice once I knew that I needed a framework. The batteries included nature meant that I was getting everything I needed in one place, pre-integrated for me, with no omissions that might have pulled me out to look at anything else.

Do I feel bad about falling for Django's PR and batteries included monolith? No. Not at all. While I have some sympathies to the smaller and leaner frameworks (I'm ordinarily a smaller is better person), I feel that frameworks are special in a way. The thing about frameworks is that they are only a means to an end, ie a working web app. Almost all web apps are small and undemanding enough that the differences between good enough frameworks are not important; any one of them will do. What really matters is getting your application written, and for that the inclusion of batteries matters a lot.

Let me be blunt. For most people selecting a framework, the last thing they want to do is then try to pick out a templating system, a database ORM layer, a forms handling package, and so on. This is especially so because all of those things are what really gets the work done for most projects; without them, a framework is only a skeleton that does, well, not very much (it's less a framework and more a request/response handling library) and provides very little help for your project. This means that including batteries by default is quite important in practice (even if it may cause political difficulties). I don't think that you have to make the default choice the only option, but if you want me to pay your framework much attention you need something that's pre-chosen and ideally pre-assembled for me; doing otherwise means that your framework simply requires too much additional time when there are other options.

Even if Django had not had the PR edge that it does, even if it had simply been one among a number of equally well known contenders, I think I would have chosen it purely for the batteries included nature. I had a project to get done and Django offered me a single solution for everything I needed so I could immediately jump into building my actual app.

Explaining a piece of deep weirdness with Python's exec

In an update in yesterday's entry on scopes and what bytecodes they use I said that the special *_NAME opcodes could wind up being used in functions but that it would take an entire entry to explain when and why, and I also included a trivia bonus that I need to explain. It will probably not surprise you to know that these two things turn out to be intimately connected.

To start with, here is an altered version of yesterday's bonus trivia that adds a second oddity:

def geta():
  return a

def exec_and_return(codeobj, x):
  exec codeobj
  return x, geta(), a

a = 5
co = compile("a = a + x; x = a", '<s>', 'exec')

print exec_and_return(co, 4)
print a

This prints '(9, 5, 9)' and then '5'.

Wait, what?

Before I start trying to explain this, let's talk about all of the things that are wrong with this result. First off, if the compile() code was simply inline in place of the exec we would get an UnboundLocalError since a is used before it's assigned to. Second, if the a was instead being treated as just a plain global its global value should change and we can see that it doesn't, either while the function is running or after it finishes; at the same time, the global value of a was clearly used to compute the result. Finally, something that looks a lot like a new local a is visible in the function with the value that we expect (the same value as x).

(You can also verify that a appears in the dictionary returned by locals().)

The first thing going on here is what happens inside the compiled code as exec runs it. Recall that NAME opcodes essentially treat a variable as global until it is assigned to, at which point it becomes a local, and that compile() generates NAME opcodes. This means that in the compiled co code object, the value of a is first read from the global a but then the assignment creates a local a (in the stack frame that exec uses to run the code); it is this local a that is then assigned to x. This explains why the global a never changes value; it is never written to, despite appearances. What the compiled code really means is something like 'al = a + x; x = al'.

(This is the entire explanation for yesterday's trivia contest.)

The second thing going on is that CPython tries quite hard to let exec'd code create new local variables in functions. It does this by changing what bytecodes get generated for references to variables that aren't definitely locals. Under normal circumstances CPython decides that you clearly mean a global variable and compiles to bytecodes that use the *_GLOBAL family of opcodes to access things. However, if you use exec in your function CPython decides that such references are unclear and compiles them to *_NAME opcodes instead, since you could also be trying to access new local variables that will be created inside the code run by exec.

(Since NAME opcodes look first at locals and then secondly at globals, this will still work if you are genuinely referring to a global variable. The example contains an instance of this in our call to geta(), as you can see by disassembling the bytecode for exec_and_return.)

But just compiling such references to NAME opcodes isn't enough enough to do the job by itself. Because NAME opcodes explicitly look at the frame's f_locals dictionary, you need a real dictionary and it really holds (some) local variables in it. This means that a function that uses exec effectively has two sets of local variables; it has the known fast local variables, stored in the local variable array and accessed with FAST opcodes, and then also any additional variables that were materialized by exec'd code, stored in the frame locals dictionary and accessed with NAME opcodes (if they're accessed at all).

(Since I was just checking this: while exec does wind up creating a new frame to run the compiled code in, as far as I can tell it does not create a new frame locals dictionary to go with it. Instead it directly reuses the frame locals dictionary of the current frame. The fast local variables are synchronized into the frame locals dictionary before it's used and exec imperfectly copies changes to them back to the local variables array after the code finishes running.)

Frankly, all of this is confusing and arcane and just goes to show how much of an impact on your language there is to try to support executing arbitrary code in the scope of a function (at least in the face of various sorts of important optimizations). We are up to one bug, conditional bytecode generation with unusual semantics, and several pieces of odd weirdness.

Python scopes and the CPython bytecode opcodes that they use

Since I just got very curious about this, I want to write it down. CPython has four different families of bytecode opcodes for accessing variables, which are used under different circumstances and look in different places. They are:

• *_FAST opcodes are used to access function local variables, which have a special storage scheme. They're used in function scope for variables that are not part of a closure (including inside a closure for variables that are local to the closure alone).

• *_DEREF opcodes are used to access 'cell variables' that are used as part of the implementation of closures; these are effectively function local variables with an extra level of indirection. They're used in function and closure scope for variables that are part of a closure.

• *_GLOBAL opcodes are used to access variables that are explicitly known to be global variables or builtins, for example if you are in a function and have used global on a variable name.

• *_NAME opcodes are used to access variables in code at the module level or in a class that's being created, and are also the opcodes that compile() generates. Effectively they are CPython's generic 'access a variable somehow' opcodes.

  (Update: under some relatively rare circumstances, NAME opcodes can also be used in functions. When and why is complicated enough to call for a full entry.)

The first three opcode families look in the obvious places you'd expect. The NAME opcodes are a bit odd and to explain them I need to talk about stack frames briefly.

CPython code always runs in the context of a stack frame. Ignoring implementation details for a moment, stack frames have three namespaces associated with them; the frame's locals, the globals, and the builtins. How the NAME opcodes work is that the LOAD_NAME bytecode looks at each of these namespaces in succession and takes the first instance of the name it finds, while STORE_NAME always writes things into the frame's locals. As the comments in the CPython source code put it:

[...] the name is treated as global until it is assigned to; then it is treated as a local.

(Technically something like this can also happen with LOAD_GLOBAL and STORE_GLOBAL, since STORE_GLOBAL will not write to the builtins but LOAD_GLOBAL will look in them as a fallback.)

When code is running at the module level, the frame's local namespace is the same as the global (module) namespace, but you can still copy from the builtins namespace to the module namespace. When code is running during class creation, the local namespace is the class to be's namespace and the global namespace is the module namespace:

a = 10
print globals() is locals()
class A(object):
  a = a
  print globals() is locals()

This prints True then False.

(This adds yet another place in CPython that I know of where the left side of an assignment can be in a different namespace than the right side, updating my previous update. One day I will have found them all.)

PS: I suspect that this implementation of the NAME opcodes explains something I noticed a while ago, which is that the C level frame structure always has a real f_locals dictionary. The code of the NAME opcodes directly uses this f_locals field, so having it always valid probably simplifies the innards of the bytecode interpreter. In fact if f_locals is NULL, the NAME opcodes throw an exception.

(I'm not certain if it's ever possible to have a NULL f_locals under normal circumstances, although if you do a web search for 'no locals found when storing', 'no locals when loading', or 'no locals when deleting' you will get some hits.)

Sidebar: A bonus trivia contest with compile() again

If you've managed to follow my writing above, you should now be able to understand and explain what happens in the following code (a variant on yesterday's weirdness):

def exec_and_return(codeobj, x):
  exec codeobj
  return x

a = 5
co = compile("a = a + x; x = a", '<s>', 'exec')

print exec_and_return(co, 1), a

This prints '6 5'.

(This behavior is probably not strictly speaking a bug and it would be hard to fix given the constraints on compile().)

(Update: the explanation is now here.)

Into the depths of a Python exec scope-handling bug

I can rarely resist a Python mystery, so when I saw The exec Statement and A Python Mystery (via Hacker News) my eyes lit right up. Here is the code in slightly shortened form (from here, via HN again):

def exec_and_return(codeobj, x):
  exec codeobj
  return x

co1 = compile("x = x + 1", '<s>', 'exec')
co2 = compile("del x", '<s>', 'exec')

print exec_and_return(co1, 1)
print exec_and_return(co2, 1)

I'll save you running this in the interpreter and tell you the answer; this prints 2 (what you'd expect) and then 1 (not what you'd expect, you'd expect an error since x is being referenced after having been deleted).

The deep answer to what is going on here is that CPython's exec imperfectly handles running code in a function scope, or specifically in matching up the scope between function scope and the compiled code's scope. The difficulties are ultimately caused by how CPython implements function scope (and accesses a function's local variables) at a low level.

Used together this way, compile() and exec are specified to take code, compile it generically (without you having to specify what sort of scope it will run in), and then have exec run that generically compiled code in the current scope, whatever that is. exec can be called on to run the same compiled code object at the module ('global') level, in a class being created, or inside a function or a closure, and it must make all of them work. In most language implementations this would be straightforward because all of these different sorts of scopes would be implemented the same way (with the same internal data structures and the same way of accessing variables). With a uniform implementation of scopes, exec and the compiled code wouldn't really care just what sort of scope it was running in; everything would work the same way regardless. Unfortunately for us, CPython doesn't work this way.

In CPython, function local variables are stored and accessed in a completely different and much faster way than variables are in all other scopes. Everything except functions holds variables in a dictionary (more or less) and accesses them by name; functions store local variables and function arguments in an indexed array and accesses them by their index number. This difference is reflected in the compiled code, where code in a function uses completely different bytecode operations to access variables than all other code does. This means that code compiled for a non-function context can't directly use a function's scope and vice versa (and in fact code compiled for one function can't use another functions's scope, because even if they have variable names in common the array offsets are probably going to be different).

(Closures can use yet another set of bytecodes with somewhat peculiar effects.)

Because it basically has no choice, compile() creates code objects that are natively designed to run in a non-function scope; they access and manipulate variables through the name lookup bytecodes (you can see this by doing 'import dis; dis.dis(co1.co_code)'). This means that when exec runs a compiled code object in a function context, it must somehow fix up the difference between the array-based function scope that actually exists and the name-based scope that the compiled code expects. So what exec does is that it materializes a fake name-based local scope (which is very similar to what you get from locals()), feeds it to the compiled code object as the local scope, and when the compiled code object finishes it copies all of the changes from the fake local scope object back to the real function scope.

Except, of course, exec has a bug; it doesn't copy all of the changes, as we've seen here. exec will copy back changes in variable values (well, their bindings) but it doesn't notice when variables have been deleted. So co1 changes the value of its local x to 2 and this change is copied back to the function scope, making the function return 2, but when co2 deletes x this deletion is not copied back; the function scope x is left untouched at 1 and is then returned.

(This bug is not intrinsic to how CPython allows function local variables to be manipulated, but fixing it would probably be complex.)

Sidebar: why compile() has no choice

The short answer is that using bytecode that's designed to run in a non-function scope is the only way that compile() can create code objects that can at least sometimes run without fixups. Even if compile() used the bytecodes for accessing a function's local variables, it doesn't know the proper array offsets for the various names; in fact, the proper array offsets will vary between functions. Consider:

def exec2(a, b, co):
  x = a + b
  exec co
  return x

In our original function, x is in the second local variable slot. In exec2, x is in the fourth local variable slot. Executing co would thus need some sort of indirect mapping table for the slot offsets in order for it to manipulate the right variable. And of course exec would have to make up a fake local scope and array if co was run in a module context.

Worse, the fixups needed in a module context would be quite challenging because the compiled code run by exec can itself make calls to outside functions. When the code called out to any outside function, exec would have to quickly copy any changes from the fake local array back to the modules's scope, call the outside function, then copy changes from the module's scope back into the fake local scope array. Let's just say no to that idea.