Why metaclasses work in Python
I've covered what you can do with metaclasses (1, 2, 3, 4) and even, sort of, the low level details of how they work (1, 2, 3). But I've never covered the high level view of why metaclasses work, ie what overall Python features make them go (partly because I am so immersed in Python arcana that much of that stuff feels obvious to me, although I doubt it actually is).
To start with, in Python everything is an object and all objects are an instance of something (yes, there are spots where this gets recursive). This includes even things that you wouldn't normally think of as objects, such as functions. Crucially, this includes classes: classes are objects. Any time you have an object in Python, a lot of its behavior is usually provided by whatever it is an instance of (to avoid confusion, I'll call this the type of the object). Classes are no exception to this; a lot of how classes behave is handled by their type, even things like how a new object gets created when you call the class.
(For simplicity, I'm going to ignore old-style Python 1.x classes from
here onwards and assume that all classes are new-style Python 2 classes
that ultimately subclass
To avoid a point of confusion: classes have ancestor ('base') classes
that they inherit from (or just
object(), the root class). However,
classes are not instances of their base class; we can see why this
has to be when we note that a class can inherit from multiple base
classes. You can't be an instance of several different things at once.
So classes exist in a two-dimensional relationship; they inherit from
one or more base classes, and at the same time they are instances of
something that provides much of their 'class' behavior. The type
of classes (the thing that provides the 'class' behavior) is called
(This two dimensional structure can get a bit weird.)
In some languages, the creation of classes is black magic that happens
deep in the interpreter and isn't something you can do inside the
language (even if the classes are visible as objects). Python has
instead chosen to expose the ability to create classes by hand; you
can do this by calling
type() with the right arguments (and then
binding the class object to a name), just as you
create instances of normal classes by calling the class itself. As part
of creating classes yourself by hand, you can obviously manipulate
class creation; you can create a new class with whatever methods, base
classes, and so on you want.
(What's odd about
type() is that despite it being a class, you can
call it with a single object to get the type of the object.)
Python is also an unusual language in another way; in Python, things
like defining functions and classes are themselves executable
statements. Python doesn't parse your program,
create all the functions and classes, and then start running your code;
instead it starts running your code and things like
execute on the fly (as does
import and so on). So it's natural to have
your code running as classes are being created.
The combination of these two things means that Python can easily provide
a way to hook your own code into the process of creating the class
objects for classes that are written in straight Python, with '
X(object): ....'. Python is already running code in general when this
happens, and the mechanisms of creating classes by hand means it's
relatively easy for Python to hand you the bits of the class-to-be so
you can modify it and then have everything continue onwards to create a
new class. This is why metaclasses can change classes as they are being
The other half of why metaclasses work is that Python allows classes to
be instances of something other than
type(). Since classes get a lot
of their 'class' behavior through normal instance method inheritance
type(), a class being an instance of something other than
type() lets the other thing intercept or change the normal as-a-class
behavior for that class (for example, what happens when you call the
class). This is why metaclasses can do things with a class after the
class has been created.