Wandering Thoughts archives

A limitation of Python properties

A periodically annoying limitation of Python properties on instances is that you cannot remove or replace a property. (You can implement del in your property, but this is not the same thing.)

This restriction exists because properties are actually attributes on the class, not attributes on the instances. Because of how attribute lookup works, you can't even sneak a replacement in through the instance's __dict__.

This means that you can't use properties to implement a highly efficient 'generate once and then cache the answer' instance attribute, where the first time you access obj.x the value gets generated and thereafter access to it goes at plain instance attribute speeds. This matters, because property access is about three times slower than accessing a simple attribute (and a plain function call to a lambda is about twice as slow).

If you need this, the best way is to make a __getattr__ function that creates the variable the first time it's accessed. This is somewhat less elegant and more monolithic than a property approach, and has the other downside that pychecker will probably complain about access to those magically created variables.

Sidebar: an interesting gotcha in timing attribute access

I did my timing of various attribute access cases using single-letter variable names. To my surprise, there were significant performance differences between the same access method based on what letter I used for the variable name. I suspect that this is because of dictionary hash collisions.

I also accidentally discovered that accessing class attributes is slightly faster than accessing instance attributes (and I don't think this one is dependent on variable names).

Please leave IOError and OSError alone

From Lawrence Oluyede's blog entry about recent Python SVN updates:

• urllib raises IOError if the servers response contains an invalid HTTP status line.

Augh. Wait, that's not strong enough, let me try again: AUGH.

In fact, it gets worse. To quote from the urrlib2 SVN source code's comment about its URLError error class:

URLError is a sub-type of IOError, but it doesn't share any of the implementation. [...]

(urllib2 also directly raises mis-formatted IOError exceptions for some things.)

Python's IOError and OSError have a very specific purpose; they report low level operating system errors. That is the only thing they should be used for, most especially if you are not going to carefully mimic the existing exception object format (which you can't do, because you don't have a real errno value). Having an alleged IOError object that looks nothing like real IOError objects is actively offensive, especially when it is in the standard library.

What these people have done, whether they realize it or not, is to turn IOError exception objects into nothing more than strings, because these modules have insured that there is no higher structure that you can count on an IOError instance having. Unfortunately the relevant Python documentation does not mention this, and continues to mislead the innocent into believing that when they catch an IOError exception they can do useful things like look at its errno or strerror attribute.

Or, to boil it down: never raise or subclass IOError, OSError, or EnvironmentError.

(This applies to EnvironmentError as a whole because using it in an except clause is the common way to catch both IOError and OSError at once.)

The only exception is if you are somehow directly calling a C library function (that is documented as setting errno on failures, which not everything does), in which case you should do the research to exactly mimic how the Python core code does it.

To my horror, urllib2 is hardly the only offender; there are even some in C level modules (in bz2module.c and zipimport.c). But that doesn't excuse them making the situation worse; in fact they should be moving towards making the situation better. Even with backwards compatibility concerns they could have raised URLError instead of IOError directly (or taken the opportunity to introduce a proper error class if they felt URLError was inapplicable).

(This rant has been brought to you by my attempt to catch up on Planet Python.)