Wandering Thoughts: Recent Entries For 2011/04/19

Another reason to avoid using __slots__ in your Python classes

Presented in the traditional illustrated form:

class A(object):
     __slots__ = ('a',)

class B1(A):
     __slots__ = ('b1',)

class B2(A):
     __slots__ = ('b2',)

Now try to define a class C that inherits from both B1 and B2, and you will get:

TypeError: Error when calling the metaclass bases
   multiple bases have instance lay-out conflict

It's relatively intuitive to see why and how this conflict happens at an abstract level. Imagine that __slots__ fields are sort of like function local variables and go in an indexed array. Then class A uses the first array slot for a, and class B1 and B2 both use the second array slot for b1 and b2 respectively. When you get to C, you have a conflict; the second array slot is used twice.

It is apparently popular in certain circles to use __slots__ just to make it so that class instances can't have extra attributes added to them at runtime. The example here illustrates the problem with this; using __slots__ has effects on what you or other people can do later with the classes.

Sidebar: about those variable names

Update: I am wrong about what I wrote below, because it turns out that a little knowledge is a dangerous thing. Per the Python data model documentation on slots, a class without a __slots__ definition turns off slots entirely, so the example below isn't doing what I think it's doing. Oops.

As it turns out, you can't avoid this conflict by giving the instance variable the same name in B1 and B2. If it really is the same variable, you need to introduce a parent B class that defines it:

class B(A):
     __slots__ = ('b',)

class B1(B):
     pass

class B2(B):
     pass

Then you can define a class C that inherits from both B1 and B2.

A difference between Python 'structs' and C structs

I've written before about how I emulate C structs in Python. This is only an emulation, and one of the differences between the two, one that is sometimes important, is that elements in such a Python struct do not have an order.

Where this comes up for me is that every so often I want to use introspection of a Python struct to automatically map between the Python struct and some flat ordering of the fields, such as a file or a network protocol message. With no inherent order to the fields, I wind up having to specify the order explicitly in some way, which always vaguely annoys me.

I consider this subtly different from my case of lists with named fields, although I could use the latter to deal with this problem. To me, lists with named fields are lists first and have named fields later, whereas these are structs with named fields that happen to (sometimes) have a canonical ordering. Possibly I am thinking too hard about this.

Of course, if you have full freedom you can always force a canonical ordering on a given Python struct if you want. Since you have the field names, you can just sort them alphabetically and declare that the canonical ordering. It may not look very nice, but it does answer the question.

(It will also cause you heartburn when you add a new field and you want it to go at the end of the ordering but of course it doesn't sort that way. Pretty soon you start manipulating the result of sorting the field names and that way lies doom and more doom.)

Sidebar: why this lack of ordering exists

My Python emulation of structs just uses a class, well, instances of a class. Instance elements (and class elements) are unordered in most circumstances because all Python does is shove them in the class or instance dictionary, and dictionaries are themselves unordered. I believe that Python classes that use __slots__ do have an actual ordering to their instance elements, but I have never looked very deeply into how that works.

The necessary disclaimer is that this is how CPython works. Other implementations of Python might give elements an actual order and just fake class and instance dictionaries for you if you look at them.

(Python dictionaries are unordered in CPython because they are implemented with hash tables, and hash tables do not have a predictable or stable ordering.)