Some notes on __slots__ and class hierarchies

June 11, 2011

How __slots__ interacts with class inheritance is a little bit complicated (although in general everything is covered in the official documentation). There are a couple of issues.

First, once a class has a dictionary, it's permanent; you cannot take the dictionary away in subclasses. This means that if you have a class hierarchy involving slots-using classes, all of their parents need to not have a dictionary, ie use __slots__ without including "__dict__".

(By the way, it's just struck me that this is fundamentally why you can't add attributes to instances of object(), as I noticed a long time ago. To have arbitrary attributes added, object() instances would have to have a dictionary, but that would mean that you couldn't subclass object() to get a dictionary-less class.)

However, you can relocate a dictionary-using parent class's instance variables into slots (which may save you memory), because slots don't have to be defined in the same class as where they're used. To make this clear, suppose that you have:

class A(object):
  def __init__(self):
    self.a = 10

class B(A):
  __slots__ = ('a',)

An instance of A will have a __dict__ with 'a' in it, but an instance of B will have an empty __dict__; a is now stored in a slot.

Using this very much will probably make everyone's head hurt, especially if you start introspecting things. (Introspection in the face of slots is one of those interesting topics. For example, there is no simple way to see what attributes an object currently has.)

Since a class normally has a dictionary, inheriting from a slots-using class without specifying __slots__ yourself results in your instances having dictionaries, as I discovered. As with the example above, the actual instance variables from your parent class will still wind up in slots. It is easy to miss this.

It's certainly possible to combine slots with having a class hierarchy, but it's clearly complex (and has limits). I would avoid it if possible, partly because some failure modes are basically silent (your instances have dictionaries when you didn't expect them to; nothing breaks, but you use more memory than you should). The simple approach is the easiest; just have all your slots-using classes be subclasses of object(). Among other things, this makes it easy to be sure of exactly what slots behavior you're getting, because the class itself is the only thing that matters.

PS: I'm sure that someone, somewhere, has written a 'slotizer' metaclass that analyzes the __init__ function in a new class to determine all instance variables that it sets and then turns them all into slots.


Comments on this page:

From 24.20.36.83 at 2011-06-12 03:57:05:

Might want to take a look at http://www.pkgcore.org/gitweb/?p=snakeoil.git;a=blob;f=snakeoil/test/test_slot_shadowing.py ; that's a testcase I add to my personal code for spotting __slots__ missuse (shadowing, having __slots__ in class a, but switching back to __dict__ in derivative class b, etc).

Either way, slots should not be avoided- you just have to pay attention to what you're doing with them. Frankly if you're generating multiple thousands of instances of a class in memory, the memory savings from slots can start adding up quickly- especially if the instances are just storing an attribute or two (python dict over-allocation for small numbers isn't great).

~ferringb

Written on 11 June 2011.
« What __slots__ are good for (or, how to use slots to save memory)
Why [], {}, and () are cheap in Python »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sat Jun 11 01:20:20 2011
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.