Python's attribute lookup order

September 23, 2011

One of the things that surprised me recently is that the Python language reference doesn't seem to document the specifics of how Python searches for the attribute when you try to access obj.attr. So here is a not completely formal description of the process for Python 2.7 or so for new-style classes.

(Such a description really belongs in the language reference because it's an important part of Python's semantics. It's possible that you can reassemble a full description of the lookup semantics from the bits and pieces scattered throughout the language reference.)

First, any time Python looks for an attribute on an object it doesn't just search the object's (nominal) attribute dictionary; it also searches back through the object's base classes (in method resolution order), ultimately terminating at object. Ordinary instances of classes don't have bases (and can't be given them), but classes do.

Let's say we're looking for attr on the object obj, and typ is the object's type object (ie type(obj)). Then the lookup order is more or less:

  1. if typ.attr exists and has both a __get__ and either a __set__ or a __delete__ attribute, it's a data descriptor; we call it and return the result.

    (Data descriptors thus deliberately preempt attributes on the object itself.)

  2. search for obj.attr. On classes (but not on non-class instances) this may be a descriptor with a __get__ instead of a plain attribute; if it is, it's called.

    (Classes evaluating their own descriptors is necessary in order to make various descriptor and property things work when you directly access the class. Consider cls.classmethod(), for example.)

  3. search for typ.attr. This may be a plain attribute or a non-data descriptor with a __get__; if it's a descriptor, it's called.

(As you might guess, the code does not search for typ.attr twice; it saves the result of the step one search for use in step three.)

Note that Python never looks at the type of typ; one level into the type hierarchy is as far as it goes. By contrast it will go as far along the base class inheritance hierarchy as it needs to, all the way to object if necessary.

(This explains how metaclasses can be used to add class-only attributes on classes. instance.attr will search only the instance and cls, its class, but not go the next step to type(cls), the metaclass. cls.attr will search cls and its base classes (in step 2) and then the metaclass (in step 3).)

It's important here to not confuse an object's base classes with its type or its type's base classes. I tend to think of base classes and types as creating a two dimensional structure, where inheritance goes sideways through a class's base classes to end up at object while its type goes up to end up at type. Although we often informally talk about instances of an ordinary class inheriting from things and attributes being looked up through them, this is not what is actually happening. A plain instance has no base classes; it is its type (the class it is an instance of) that has the base classes. Normally making this distinction is unimportant, but here it's vital.

(The confusing case is a metaclass, which has type as both its base class (since metaclasses subclass type) and its type (since all classes, metaclasses included, are normally instances of type).)

Sidebar: where this is done in CPython's code

In CPython, this is done in two separate pieces of code. Lookups for classes are handled by type's type_getattro(), in Objects/typeobject.c. Lookups for non-class objects are handled by object's PyObject_GenericGetAttrWithDict(), in Objects/object.c (which is called with no dictionary argument in this case).


Comments on this page:

From 188.226.51.71 at 2011-09-23 03:42:19:

It looks like you've totally left out __getattr__ and __getattribute__ special methods, I wonder why.

Guess it's true that they work "around" the process you've described, but still "obj.attr" may call either of them as a part of lookup process for attribute "attr", no?

By cks at 2011-09-23 13:32:12:

A good part of the real answer is that I don't entirely understand the CPython code that implements these two special methods, so I can't be sure that I'm getting the real semantics right. Apart from that, I decided that they weren't really part of the attribute lookup order as such, since they either replace it or come in after it's failed, and that adding them would add too much complication.

(They're also both well documented, unlike the rest of the lookup order.)

The documented semantics are that typ.__getattribute__ is called instead of this entire process and typ.__getattr__ is called if the entire lookup process fails (whether it was a call to typ.__getattribute__ or a normal lookup).

From 193.52.208.117 at 2011-12-03 09:40:10:

What about reporting this doc issue at bugs.python.org? Bug requests lead to patches that benefit all users, contrary to blog posts. —Éric Araujo

By cks at 2011-12-03 23:18:06:

Some things:

  • filing decent bug reports is a lot of annoying work unless one is already an insider in whatever system it is: one has to register, check if the bug has already been filed, check if the problem has been fixed in the in-development versions, and so on.

    Then, of course, one often gets the pleasure of either arguing with people about whether the issue is actually a bug or of seeing the bug report (and the effort one put into it) disappear into the unresponsive void.

  • filing bad bug reports has social consequences that matter if one ever wants more important bug reports to be acted on. It is better not to file at all than to file a bad bug report.

  • whether or not you realize it, leaving comments on blog entries to suggest that people file bug reports is insulting; you are implying that they are so stupid that this idea hasn't already occurred to them (and been passed over for some reason that they find convincing). Backhanded slams about the worth of their blog entry do not help.

In my opinion the delicate and polite way to encourage people to file bug reports in these situations is to leave a comment saying roughly 'you're got a good point; I've created a bug report on your behalf (at <url>) so that we can track this and fix it, in the future you can register at bugs.python.org and submit these sorts of things yourself'.

(Adding 'this sort of thing is something we do want to fix' is optional but might be useful for things like documentation changes, since it is not always obvious if projects, say, actually want the current semantics of something to be carefully documented or consider it implementation dependent.)

Written on 23 September 2011.
« An operational explanation of Python types
Some recent Google spam problems »

Page tools: View Source, View Normal.
Search:
Login: Password:

Last modified: Fri Sep 23 00:30:31 2011
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.