Wandering Thoughts archives

2013-09-13

Why I think dir() excludes metaclass methods

I was recently reading David Halter's Why Python's dir function is wrong which points out that dir() on classes and types excludes some valid attributes from the result (for example, __bases__). As it happens, I have a theory about why Python behaves this way. The short version is that it is a heuristic to make dir() more useful.

(Note that classes and types are the same thing. From now on I'm going to use 'class' to mean both.)

When you use dir() on a class there are at least two things you can be interested in, namely what attributes will be visible on an instance of that class (ie, what attributes are defined by the class and its inheritance hierarchy) and what attributes are visible on the class itself (more or less). My feeling is that almost all of the time people use dir() they are more interested in the former question; in fact, I'd expect that many people don't even know that you can have attributes visible on a class but not its instances.

Even ignoring the direct usability issue, doing dir() 'properly' has a couple of other significant drawbacks. First, you lose any good way to find out what attributes will be visible on instances of classes; you'll wind up wanting to add a special flag to dir() to return to the old behavior. Second, the result of dir(anyclass) is likely to be confusing in practice because it will mingle instance-visible and class-only attributes, including reasonably common special methods. Most obviously, every class has a __call__ special method from type() but it can only be called on the class itself.

It's probably worth mentioning the obvious thing, namely that metaclasses didn't used to exist while dir() is very old. Effectively this set the behavior of dir() as excluding metaclasses; you can imagine the uproar if dir() had suddenly added a bunch of methods when used on classes (including giving __call__ to everyone) when metaclasses were introduced. This might well have been (correctly) regarded as a significant change from existing behavior.

(This seems especially likely as I believe that there is some code that uses dir() on classes for introspection.)

Also I've played fast and loose with something here, because dir() is actually not the list of what attributes will be visible on anything. dir() is the list of attributes that are defined on particular things, but additional attributes can be materialized in various ways and defined attributes can even be made inaccessible if you try hard enough. This may suggest why any change in dir()'s behavior has not been a high priority for Python developers; in a sense you usually shouldn't be using it in the first place. And (as pointed out on reddit) this dir() behavior is explicitly documented and it does even more magic than this.

PS: for more on metaclasses and many low level details about them, see my index to my metaclass series.

Sidebar: what additional attributes you would see on normal classes

You can get this list in a simple way:

sorted(set(dir(type)) - set(dir(object)))

Rather than put a big chunk of output here I think I'm going to leave it to my readers to run this for themselves if they're interested.

Classes with custom metaclasses would add any additional attributes from those custom metaclasses.

python/OnDirAndMetaclasses written at 01:14:39; Add Comment


Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.