Wandering Thoughts archives

Function definition order in Python

Thomas Boutell has just started serious Python work and wound up noticing (in the middle of here):

What's not so painless is discovering that "function" must be defined like this before I can call it.

Wait a minute. That's pretty standard behavior for C, yeah. But this ain't 1973. Perl and Java are both bright enough to read the rest of your code and find the function, allowing you to place functions in your source code where they feel right to you.

So: why is Python this way? The answer's deeper and more interesting than it might look.

There are three things going on here, and they all make sense when you understand them:

1. Functions are found by name lookup, like pretty much everything else in Python. So the name has to be defined by the time something tries to call the function.
2. Doing an 'import <module>' executes the file's code; the same thing happens when you do 'python file'. (What's left in the namespace after the code finishes running basically is the module.)
3. def is an executable statement; when it runs it creates the name and points it to a function object, which has the function's bytecode and some trimmings. (class is also an executable statement.)

(Technically import only executes the file's code the first time around. After that, it looks up the module in an internal table and horks back a reference.)

Understanding the third is actually quite important, because it is how and why a great many clever tricks with classes work (including straightforward uses of staticmethod and classmethod decorators). Consider:

class Foo:
  def bar():
    return 42
  bar = staticmethod(bar)

This works because everything in the class statement is actually being executed. The def creates and binds bar's value to a function object, and then the next line rebinds bar's value to a different object created by staticmethod. When the dust settles, the class's namespace has only the new bar.

(staticmethod itself is an ordinary function (well, a type); you can run it outside a class definition if you want to, although the objects it creates are not really useful outside of classes.)

This is also why function closures work, because they are recreated with the right bound variables every time around. Consider:

def foo(a, b):
  def bar(z):
    return a + b + z
  return bar

Although this doesn't literally reparse bar's source each time, it does make a new version of bar on each call. (You can see this by using the dis module to look at the generated bytecodes.)

So Python is not strict 'define before reference', the way a language like C is, but 'define before use'. You can have function definitions in any order and in any place, so long as they're all defined before any executing code tries to use them. However, if you put code at the top level of a file (where it will get run at import time) the functions the code uses must be lexically before the code.

This approach to function definition is by no means unique to Python. LISP was probably the first language to do it, but you can find this in lots of others too.

Because there's nothing magical about function names, there's a number of tricks that can be played with them. For example, as we saw with 'class Foo', function names can be rebound to other values. You can also create new functions just by binding an appropriate value to a name.

(One use of this is to decide on the fly which version of code will implement a generic interface; an extreme example is the os module (look for os.py). os.py is also a good illustration how far you can go by running code during import.)

(This entry is adopted from my LiveJournal comments on here.)