How you access an object can be important in Python

April 29, 2010

One of the less than obvious things about Python is that it can matter a lot just how you access an object, and not just for performance reasons. Accessing the same object through two different names can have drastically different results under some circumstances, as yesterday's entry about frame.f_locals shows.

(This is especially the case when resolving one of the names involves a C module (instead of just a Python one). C modules generally need 'getter' functions to translate their C-level data structures into the Python representation of them, so there's a greater temptation to make the getter functions do something clever since they already exist.)

The fundamental thing happening is that when you look up attributes on an object, you give that object an opportunity to hijack your lookup and do something clever (or at least something that it thinks is clever). This happens every time you perform an attribute lookup, because Python does not cache them (and cannot in the presence of any of the various attribute lookup interception functions). Thus, every time you access thing.attribute you can get a different result, or you can get the same result but something is done to it behind your back (as was the case with frame.f_locals).

(I checked and the frame.f_locals getter does always return the same dictionary object, it just mangles it every time.)

Obviously, this can lead to very counterintuitive or outright nonsensical results, ones that generally take reading the source code of the object involved to understand. (As handily illustrated last entry.)

Note that this is different from returning objects that are themselves magic. Again, frame.f_locals is an excellent example; what it returns is a standard dictionary, so everything you do with the actual dictionary is perfectly normal. It is just that every time you do an attribute lookup on frame.f_locals to get that dictionary object, something special happens.

As far as I know, the rules on what can be intercepted this way are:

  • local and global name lookups can't be intercepted at all.
  • module name lookups (ie, accessing 'module.thing') for Python modules can be intercepted if you try hard but this is vanishingly rare.
  • object and class attribute lookups can easily be intercepted.
  • all lookups involving C-level objects and modules can easily be intercepted and often are, because there generally has to be some C-to-Python translation done anyways.

(Technically you might be able to do something excessively clever with lookups in your own module's global namespace, but if so hopefully you know about it. Because of how they are implemented, local variable lookups are truly un-interceptable.)

My intuition is that C modules are more likely to do special magic on attribute access than Python modules, because in C it is easier to mutate a standard Python object just before you return it than it is to return an entirely custom object with custom behavior. In Python code it's more even, so you're more likely to see such magic done through objects with customized behavior.

Written on 29 April 2010.
« Altering a Python function's local variables with a trace function
Never kill the screen locker »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Thu Apr 29 02:08:10 2010
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.