Wandering Thoughts archives

How CPython implements __slots__ (part 2): access

In the first part I covered how the __slots__ instance attributes were stored (in an ad-hoc array of pointers that is glued on the end of instance objects), but that's only half of the puzzle. The other half is letting people access them, and that's what this entry is about.

As it happens, it's reasonably common in CPython for C-level types to want to give people Python-level access to fields in C structures, common enough that the CPython infrastructure for new-style types built an entire set of machinery to support it. To use this machinery you specify the names, (C-level) types, offsets, and optionally some flags like 'read only', for all of the fields that you want people to see and the type-registration process takes care of all of the rest for you. One example of where this is used is the __dictoffset__ attribute of types, which directly reads the tp_dictoffset field in the C-level type structure.

(I personally find this to be a very cool feature because it avoids a whole lot of potential code duplication in C-level stuff and makes it very easy to expose interesting fields.)

The Python-level manifestation of this machinery is member_descriptor objects. Like many things in CPython 2.x, they're complicated by some backwards compatibility issues with the older C API for CPython extension modules, so I am going to pass over the fine details and just say that attempts to read or write these fields winds up in a big switch statement that checks the type of the field and does the right magic to read or change it. There are all sorts of field types supported by this machinery; one of them is T_OBJECT_EX, where the field is a pointer to a Python object and an exception is raised if the pointer is NULL.

You know, that sounds a lot like how __slots__ attributes behave. That is not a coincidence.

How __slots__ attributes are accessed is that during the process of registering the type that is the new slot-using class, the core type machinery dynamically creates appropriate name and offset information for each attribute slot and registers them exactly as if they were conventional fields in the basic C structure of the type. This name and offset information is glued on the end of the basic type C-level structure, which is why type has a non-zero __itemsize__; each 'item' is a field member registration structure.

(As a trivia note, I believe that this makes type the only standard type where using __slots__ in a subclass has absolutely no effect. Since it has a non-zero itemsize you can't use a non-empty __slots__, and since it has a non-zero dictoffset an empty __slots__ will not stop your subclass from having a __dict__.)

PS: all of this is for Python 2. I haven't looked at the CPython source for Python 3.x.

Sidebar: where to find this in the CPython source

type itself is in Objects/typeobject.c. Member descriptors are one of the classes of descriptors in Objects/descrobject.c. Actually reading and writing structure members is done by code in Python/structmember.c.

The member list of a type is pointed to by its tp_members field; this is the basic raw form, not member descriptors. Member descriptors are not created immediately on type registration; instead this, and a lot of other type setup, is deferred until PyType_Ready() is called, which is done any time the code needs to look at various type information like, say, the attribute dictionary.

Some notes on what __dictoffset__ on types means in CPython

I mentioned __dictoffset__ in passing in HowSlotsWorkI. Today I feel like expanding on that passing mention with some notes. All of this is specific to CPython.

As mentioned in passing in HowSlotsWorkI, the __dictoffset__ attribute tells you the offset to where you find the pointer to the __dict__ object in any instance object that has one. It is in bytes. A positive value is an offset from the start of the object; a negative value is an offset from the end of the object, and is used only for classes derived from types (such as str and long) that have a variable-sized component. A __dictoffset__ value of zero means that the type (or class) doesn't have a __dict__ attribute.

(You can tell which types have a variable-sized component by looking at their __itemsize__ attribute; zero means that they don't have such a component.)

As sort of discussed in the sidebar in HowSlotsWorkI, if you inherit from something with a zero __dictoffset__ your subclass will normally have a non-zero __dictoffset__ and the pointer to the __dict__ object will be glued on the end of the C-level blob of your basic type.

Most built-in types have a __dictoffset__ of zero, as you'd expect. However, a few types have a non-zero __dictoffset__; the ones I know of are exceptions, functions, modules, and type itself. What is going on is that all of these types already have to have some sort of dictionary for their contents, along with a pointer to this dictionary in their basic C-level blob. So they reuse this pointer (and associated dictionary) as their __dict__, by pointing __dictoffset__ directly to this internal field. One consequence of this is that subclasses of these classes always have a __dict__, even if your subclass uses __slots__.

(In general, once a class has a non-zero __dictoffset__ all of its subclasses will always have a __dict__. I think that you can sometimes still save space and allocations by using __slots__, but you don't get any of the other features of __slots__ that people are sometimes unwisely attracted to.)

Nailing down new-style classes and types in Python

Since I keep confusing myself, it's time to write this stuff down once and for all to make sure I have it straight (even if some or all of it is in the official documentation).

One writes Python code to define classes; it's right there in the language syntax, where you write 'class A(object): ...'. Defining a class creates a type object for that class, which is an instance of type; this C-level object holds necessary information about the class and how it's actually implemented. This type object is what is bound to the class name; if you define a class A, 'type(A)' will then report <type 'type'>.

Classes have a class inheritance hierarchy, which is ultimately rooted at object (including for C-level classes). However, strictly speaking there is no type hierarchy as far as I know; all types are simply instances of type (including type itself). Further, the type non-hierarchy is of course unrelated to the class hierarchy. This means that isinstance(A, type) is True but issubclass(A, type) is both False and the wrong question (unless you really do have a subclass of type somewhere in your code).

(Among other things I believe that this means that 'type(type(obj))' is always 'type' for any arbitrary Python object, since all objects have a type and all types are instances of type.)

The Python documentation sometimes talks about a 'type hierarchy'. What it means is either 'the conceptual hierarchy of various built-in types', such as the various forms of numbers, mutable sequences, and so on, or 'the class inheritance hierarchy of built-in types', since a few are subclasses of others and everyone is a subclass of object.

(Some languages really do have a hierarchy of all types, with real (abstract) types for things like 'all numeric types' or 'all mutable sequence types', but Python does not. You can see this by inspecting the __mro__ attribute on built in types to see the classes involved in their method resolution order; the MRO of a type like int is just itself and object. Only a few built in types are subclasses of other types.)

PS: yes, almost all of this is in the Python documentation or is implied by it. Writing it down anyways helps me get it straight in my own head.

PPS: I believe that technically it would be possible for a sufficiently perverse extension module to create a valid new style C-level class that was not a subclass of object. Don't do that, and if you did I expect that things would blow up sooner or later.

Sidebar: the real difference between classes and types

If you use repr() on user-defined classes and on built in types (eg 'repr(A)' and 'repr(str)'), you'll notice that it reports them differently. This is a bit odd once you think about it, since they are both instances of type and so are using the same repr() function, yet one reports it is a 'class' and the other reports it is a 'type'.

In CPython, the difference between the two is whether the C-level type instance structure is flagged as having been allocated on the heap or not. A heap-allocated type instance is a class as far as type.__repr__() is concerned; a statically allocated one is a type. All classes defined in Python are allocated on the heap, like all other Python-created objects, and so report as classes. Most 'types' defined in C-level extension modules are statically defined and so get called types, but I believe that with sufficient work you could create a C-level type that had a heap allocated type instance and was reported as a class.

(It's easy enough to keep it from being garbage collected out from underneath your extension module; you just artificially increase its reference count.)

How CPython implements __slots__ (part 1): storage

At an abstract level, each instance of a conventional class has a __dict__ member that is a conventional Python dictionary, and instance attributes are created and manipulated by manipulating this dictionary; the dictionary key is the attribute name and the value is the attribute's value. __slots__ eliminates this dictionary and instead has a fixed list of attributes that instances of the class know about. All of this is in the documentation. What the documentation won't tell you is how the machine level storage for all of this actually works. That's what today's entry is about.

In CPython, class instances start out as a more or less opaque C structure that is specific to the C-level type that your class inherits from (we saw this before). However, the general CPython type infrastructure for new-style classes reserves the right to add some extra space on the end of your type's opaque blob for its own purposes. If your class has a __slots__, this code adds some extra space after the C structure blob to store what is effectively an array of pointers to Python objects. These entries are used to point to the values of each __slots__ attribute (if there is no value set, the corresponding entry is NULL and the CPython code reacts appropriately).

While somewhat complicated, this approach minimizes the memory overhead for class instances. If you allocated the array of slot value pointers separately, you would have a second memory allocation and you'd need an extra pointer in the base object structure to point to the separate array. And because all instances of the class have exactly the same slots, you can put all information on the names of slots and how to access them on the class, instead of having to have it also attached on the instance.

If you have a class that both has a non-empty __slots__ and tries to inherit from certain built in types, you will get the error:

nonempty __slots__ not supported for subtype of '<type>'

The Python documentation mentions this but does not explain the details of what is going on, which have to do with this storage approach.

Most C-level types have a fixed size C structure; however, the type infrastructure has general support for types that have a fixed size header structure plus some number of (fixed size) items immediately after the header. Because the information on how to access slot values is attached to the class, not the instance, the CPython code requires that all slot value pointers have a constant offset from the start of the instance object. This requires that all instance objects for a type have the same fixed size, which is not the case for instances of 'base + items' C-level types. Hence the message you get here.

You can still have an empty __slots__ even for 'base + items' types, because this doesn't require allocating any slot value pointers; it just turns off the creation of the __dict__ dictionary.

(Well, usually.)

Sidebar: how __dict__ itself is (usually) implemented

One might innocently think that __dict__ would be implemented by having something like an ob_dict pointer in the basic Python C-level object structure. As it happens, CPython is both more clever and more sleazy than this. The storage for the pointer to the __dict__ dictionary is actually usually created through this same 'add things on the end of the type's blob' code, and the C structure for the type itself has a field that says what offset this pointer is to be found at. This saves a pointer when __slots__ turns off __dict__ and probably has other implementation advantages that I don't know about.

You might wonder how this works for base + items types. That's where the sleaze comes in: CPython has special magic support to make this work for the __dict__ offset. If I'm reading the code right, it switches to indexing the offset from the end of the object instead of the start.

(If you want the gory details, see _PyObject_GetDictPtr in Objects/object.c in the CPython source code.)

If you want to see some of this sausage's insides, look at the __dictoffset__ attribute of any new-style class. For bonus points, create a class that inherits from, say, str and then look at its __dictoffset__. Note that almost all built-in types will show a 0 for this value for reasons that do not fit into this sidebar.

Another reason to avoid using __slots__ in your Python classes

Presented in the traditional illustrated form:

class A(object):
     __slots__ = ('a',)

class B1(A):
     __slots__ = ('b1',)

class B2(A):
     __slots__ = ('b2',)

Now try to define a class C that inherits from both B1 and B2, and you will get:

TypeError: Error when calling the metaclass bases
   multiple bases have instance lay-out conflict

It's relatively intuitive to see why and how this conflict happens at an abstract level. Imagine that __slots__ fields are sort of like function local variables and go in an indexed array. Then class A uses the first array slot for a, and class B1 and B2 both use the second array slot for b1 and b2 respectively. When you get to C, you have a conflict; the second array slot is used twice.

It is apparently popular in certain circles to use __slots__ just to make it so that class instances can't have extra attributes added to them at runtime. The example here illustrates the problem with this; using __slots__ has effects on what you or other people can do later with the classes.

Sidebar: about those variable names

Update: I am wrong about what I wrote below, because it turns out that a little knowledge is a dangerous thing. Per the Python data model documentation on slots, a class without a __slots__ definition turns off slots entirely, so the example below isn't doing what I think it's doing. Oops.

As it turns out, you can't avoid this conflict by giving the instance variable the same name in B1 and B2. If it really is the same variable, you need to introduce a parent B class that defines it:

class B(A):
     __slots__ = ('b',)

class B1(B):
     pass

class B2(B):
     pass

Then you can define a class C that inherits from both B1 and B2.

A difference between Python 'structs' and C structs

I've written before about how I emulate C structs in Python. This is only an emulation, and one of the differences between the two, one that is sometimes important, is that elements in such a Python struct do not have an order.

Where this comes up for me is that every so often I want to use introspection of a Python struct to automatically map between the Python struct and some flat ordering of the fields, such as a file or a network protocol message. With no inherent order to the fields, I wind up having to specify the order explicitly in some way, which always vaguely annoys me.

I consider this subtly different from my case of lists with named fields, although I could use the latter to deal with this problem. To me, lists with named fields are lists first and have named fields later, whereas these are structs with named fields that happen to (sometimes) have a canonical ordering. Possibly I am thinking too hard about this.

Of course, if you have full freedom you can always force a canonical ordering on a given Python struct if you want. Since you have the field names, you can just sort them alphabetically and declare that the canonical ordering. It may not look very nice, but it does answer the question.

(It will also cause you heartburn when you add a new field and you want it to go at the end of the ordering but of course it doesn't sort that way. Pretty soon you start manipulating the result of sorting the field names and that way lies doom and more doom.)

Sidebar: why this lack of ordering exists

My Python emulation of structs just uses a class, well, instances of a class. Instance elements (and class elements) are unordered in most circumstances because all Python does is shove them in the class or instance dictionary, and dictionaries are themselves unordered. I believe that Python classes that use __slots__ do have an actual ordering to their instance elements, but I have never looked very deeply into how that works.

The necessary disclaimer is that this is how CPython works. Other implementations of Python might give elements an actual order and just fake class and instance dictionaries for you if you look at them.

(Python dictionaries are unordered in CPython because they are implemented with hash tables, and hash tables do not have a predictable or stable ordering.)

Monkey subclassing for fun and profit

Monkey subclassing is my own term for something that I wind up doing every so often, formed by analogy to monkey patching. Where monkey patching is changing the class itself, monkey subclassing is changing the class by subclassing it and then overriding internal method functions (generally undocumented ones, or at least ones not officially exposed as public interfaces). Monkey subclassing is different from regular subclassing in that I am playing fast and loose with the internals of someone else's code and in fact these classes may never have been designed to be subclassed to start with.

Monkey subclassing has the advantage that it only changes the class for you, not for all code in your program that's using the class. It has the potential disadvantage that you may need to force other bits of the program to use your new subclass instead of the original class, forcing you to monkey subclass them in turn and soon it is monkeys all the way up.

(This depends on what sort of a change you want; sometimes you're happy with a private version of the class with a particular feature that you need.)

All of this is abstract, so let me make it concrete. Django has a few options for sending email, but for our purposes none of them were ideal; we could either send via SMTP with its problems or invoke sendmail directly, which didn't offer enough control of headers and suchlike. What we wanted to do was talk SMTP to the local mailer over a pipe, which would give us the full control we needed while meaning that we didn't have to worry about handling retries and queueing ourselves, but there is no canned solution for that right now in Django (or at least none that I saw when I went looking).

Talking SMTP over a pipe to a local process is very similar to talking SMTP over the network to a remote SMTP server, and Django mostly uses standard Python things to do the latter. So I made two monkey subclasses. First, I subclassed the standard Python smtplib's SMTP class to have a connect() method that invoked the local mailer with the right arguments in a subprocess instead of making a network connection. Second, I monkey subclassed Django's standard SMTP email backend to have an open() method that used my new monkey 'local SMTP' subclass instead of smtplib.SMTP to create its 'SMTP connection' that it would send mail over. The net result was that with two relatively short routines I had just what I wanted: our Django application sends mail using the local mailer but with full control over the message envelopes and headers.

I don't think that either class considers the methods that I overrode in my subclasses to be public interfaces. In fact, I'm not sure that either class is intended to be subclassed by outside people. And certainly a code change in either module could easily break my subclassing; monkey subclassing is the fast way, not necessarily the morally correct way.

(The morally correct way to solve my problem would be to write my own Django email backend from scratch in order to do this, and never mind the code duplication and time involved.)