Wandering Thoughts archives

Exploiting Python metaclasses to forbid subclassing and where it fails

Suppose, entirely hypothetically and purely for illustration, that you want to create a class that cannot be subclassed. Python is normally a very permissive language, but as I mentioned in passing yesterday you can abuse some metaclass features to do this. Well, to more or less do this; as it happens there is a way around it because Python allows callables to be specified as your metaclass.

While there are complex approaches with a single metaclass that has a sophisticated __new__, it's easier to illustrate the whole idea like this (in Python 3 for once):

class Block(type):
  def __new__(meta, cname, bases, cdict):
    raise TypeError("cannot subclass")

def metacls(cname, bases, cdict):
  return type.__new__(Block, cname, bases, cdict)

class A(metaclass=metacls):
  pass

This works basically by cheating; our callable metaclass gives the newly created class a different metaclass that blocks the creation of further classes in its __new__. So let's try to get tricky to get around this. Our first attempt is to give our new subclass a different metaclass that doesn't block class creation:

class MNothing(type):
  pass

class B(A, metaclass=MNothing):
  pass

This will fail:

TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

But wait, we don't actually need a new class here. We just need something that allows class creation but gives the new class the metaclass it needs in order to not have a metaclass conflict. As it happens we already have that:

class B(A, metaclass=metacls):
  pass

(note that if we didn't have metacls(), creating it is trivial. type(A) will give us the metaclass class we need to shim in.)

This not only works, this is the simple and more or less ultimate bypass for anything that tries to block __new__ because using a callable as the metaclass splits apart what happens to a class as it's being created from the class's metaclass class. The new class's official post-creation metaclass gets no chance to intervene during creation; it just winds up being the metaclass of a new class by fiat, with nothing to say about it. Since the metaclass's __init__ (if any) doesn't get called here, the first our Block metaclass can possibly know about the new class is when the new class is used for something (attribute lookup, instance creation, or subclassing).

All of which goes to show that Python is permissive after all if you know the right tricks, even if it looks restrictive initially.

(Well, at least here.)

I wish Python didn't allow any callable to be a 'metaclass'

One of the things that any number of discussions of metaclasses will tell you is that any callable object can be a metaclass, not just a class. This is literally and technically true, in that you can set any callable object (such as a function) as a class's metaclass and it will be invoked on class creation so that it can modify the class being created (which is one of the classical uses of metaclasses).

The problem is that modifying classes as they are being created is the least of what metaclasses can do. Using a callable as your 'metaclass' means that you get none of those other things (all of which require being a real metaclass class). And even your ability to modify classes as they're being created is incomplete; using a callable as your metaclass means that any subclasses of your class will not get your special metaclass processing. This may surprise you, the users of your code and your 'metaclass', or both at once. Unfortunately it's easy to miss this if you don't know metaclasses well and you don't subclass your classes (either in testing or in real code).

I understand why Python has allowed any callable to be specified as the metaclass of a class; it's plain convenient. In the simple case it gives you a minimal way of processing a class (or several classes) as they're being created; you can just write a function that fiddles around with things and be done with it; you don't need the indirection and extra code of a class that inherits from type() and has a __new__ function and all of that. It also at least looks more general than restricting metaclasses to be actual classes.

The problem is that this convenience is a trap lying in wait for the unwary. It works only in one place and one way and doesn't in others, failing in non-obvious ways. And if you need to convert your callable into a real metaclass because now you need some additional features of a real metaclass class, suddenly the behavior of subclasses of the original class may change.

So on the whole I wish Python had not done this. I feel it's one of the rare places where Python has prioritized surface convenience and generality a little too much. Unfortunately we're stuck with this decision, since setting metaclass to any callable is fully documented in Python 3 and probably can't ever be deprecated.

PS: Note that Python is actually inconsistent here between real metaclass classes and other callables, since a metaclass that is a class will have its __new__ invoked, not its __call__, even if it has the latter and thus is callable in general. This is absolutely necessary to get metaclass classes working right, but that this inconsistency exists is another sign that this whole 'any callable' thing shouldn't be there in the first place.

Sidebar: the arguments to your metaclass callable

The arguments for a general callable are slightly difference from the arguments a real metaclass __new__ receives. You get called as:

def metacls(cname, bases, cdict):
    return type(cname, bases, cdict)

If you want to call type.__new__ directly, you must provide a valid metaclass as the first argument. type itself will do, of course. Using a metacls() function that shims in an actual class as the real metaclass is beautifully twisted but is going to confuse everyone. Especially if your real metaclass has a __new__.

(If your real metaclass has a __new__, this will get called for any subclasses of what you set the metaclass function on. I suppose you could abuse this to more or less block subclassing a class if you wanted to. Note that this turns out to not be a complete block, at least in Python 3, but that's another entry.)

The technical side of Python metaclasses (a pointer)

I recently read Ionel Cristian Mărieș' Understanding Python metaclasses (via Planet Python), which is a great but deeply technical explanation of what is going on with them in Python 3. To give you the flavour, Ionel goes right down to the CPython interpreter source code to explain some aspects of attribute lookup. If nothing else, this is probably the most thorough documentation I've ever seen of the steps and order in Python's attribute lookups. There are even very useful decision tree diagrams. I'll probably be using this as a reference for some time, and if you're interested in this stuff I highly recommend reading it.

I'm personally biased, of course, so I prefer my own series on using and then understanding Python metaclasses. Ionel has a much more thorough explanation of the deep technical details (and it's for Python 3, where mine is for Python 2), but I think it would have lacked context and made my eyes glaze over had I read it before I wrote my series and wound up with my own understanding of metaclasses. But Ionel's writeup is a great reference that's more thorough than, for example, my writeup on attribute lookup order.

(But the curse (and blessing) of writing the entries myself is that I can no longer look at metaclass explanations with a normal person's eyes; I simply know too much and that influences me even if I try to adopt a theoretical outsider view.)

I do disagree with Ionel on one aspect, which is that I don't consider general callable objects to be real metaclasses. General callable objects can only hook __new__ in order to mutate the class being created; true metaclasses do much more and work through what is a fundamentally different mechanism. But this is really a Python documentation issue, since the official documentation is the original source of this story and I can hardly blame people for repeating it or taking it at its word.

PS: I continue to be surprised that Python lacks official documentation of its attribute lookup order. Yes, I know, the Python language specification is not actually a specification, it's an informal description.