Wandering Thoughts: Recent Entries For 2012/01/27

Why metaclasses work in Python

I've covered what you can do with metaclasses (1, 2, 3, 4) and even, sort of, the low level details of how they work (1, 2, 3). But I've never covered the high level view of why metaclasses work, ie what overall Python features make them go (partly because I am so immersed in Python arcana that much of that stuff feels obvious to me, although I doubt it actually is).

To start with, in Python everything is an object and all objects are an instance of something (yes, there are spots where this gets recursive). This includes even things that you wouldn't normally think of as objects, such as functions. Crucially, this includes classes: classes are objects. Any time you have an object in Python, a lot of its behavior is usually provided by whatever it is an instance of (to avoid confusion, I'll call this the type of the object). Classes are no exception to this; a lot of how classes behave is handled by their type, even things like how a new object gets created when you call the class.

(For simplicity, I'm going to ignore old-style Python 1.x classes from here onwards and assume that all classes are new-style Python 2 classes that ultimately subclass object.)

To avoid a point of confusion: classes have ancestor ('base') classes that they inherit from (or just object(), the root class). However, classes are not instances of their base class; we can see why this has to be when we note that a class can inherit from multiple base classes. You can't be an instance of several different things at once. So classes exist in a two-dimensional relationship; they inherit from one or more base classes, and at the same time they are instances of something that provides much of their 'class' behavior. The type of classes (the thing that provides the 'class' behavior) is called type().

(This two dimensional structure can get a bit weird.)

In some languages, the creation of classes is black magic that happens deep in the interpreter and isn't something you can do inside the language (even if the classes are visible as objects). Python has instead chosen to expose the ability to create classes by hand; you you can do this by calling type() with the right arguments (and then binding the class object to a name), just as you create instances of normal classes by calling the class itself. As part of creating classes yourself by hand, you can obviously manipulate class creation; you can create a new class with whatever methods, base classes, and so on you want.

(What's odd about type() is that despite it being a class, you can call it with a single object to get the type of the object.)

Python is also an unusual language in another way; in Python, things like defining functions and classes are themselves executable statements. Python doesn't parse your program, create all the functions and classes, and then start running your code; instead it starts running your code and things like def and class execute on the fly (as does import and so on). So it's natural to have your code running as classes are being created.

The combination of these two things means that Python can easily provide a way to hook your own code into the process of creating the class objects for classes that are written in straight Python, with 'class X(object): ....'. Python is already running code in general when this happens, and the mechanisms of creating classes by hand means it's relatively easy for Python to hand you the bits of the class-to-be so you can modify it and then have everything continue onwards to create a new class. This is why metaclasses can change classes as they are being created.

The other half of why metaclasses work is that Python allows classes to be instances of something other than type(). Since classes get a lot of their 'class' behavior through normal instance method inheritance from type(), a class being an instance of something other than type() lets the other thing intercept or change the normal as-a-class behavior for that class (for example, what happens when you call the class). This is why metaclasses can do things with a class after the class has been created.