Wandering Thoughts archives


What __slots__ are good for (or, how to use slots to save memory)

First off and to echo what everyone says, slots are for saving memory. The direct corollary of this is that when you are considering using slots you should measure the memory savings you get with and without them; if it is not significant (to you), you should not use slots because slots have various odd side effects and drawbacks.

(Sometimes people are attracted to those side effects. Please don't be one of those people.)

As the documentation says, slots save you memory by eliminating (or at least minimizing) the per-instance dictionary that objects of your class would normally have. How much memory this saves you is complicated (because dictionary space usage is complicated); a lot depends on how many entries your instance dictionary would normally have. There are two different but closely related things you do with slots: you can disable the creation of an instance dictionary and you can preallocate space for specific instance attributes. These things lead to three different ways of using slots.

The first way is subclassing built in types when all you're doing is redefining or adding methods. Instances of most built in types don't have instance dictionaries, but instances of your subclass normally will (even if you never use it for anything). Using an empty __slots__ will turn this off, making your instances as memory-efficient as the parent type, and you can do this even for variable-size types like str and tuple. This is the only thing you can do with slots if you're subclassing a variable-sized type, one that has a non-zero __itemsize__ (see here and here for more discussion of this).

(This also applies if you are subclassing a slots-using class. Basically, you can do this any time you're subclassing a class that doesn't already have an instance dictionary.)

Second, if you have a fixed set of instance attributes you can preallocate space for all of them and entirely eliminate the instance dictionary. This is the classical case of a non-empty __slots__ with attribute names in it. In many situations it makes sense to preallocate instance attributes that you will only rarely use so that you can get rid of the instance dictionary (and slots arrange for hasattr() et al to keep on working in this case; a slot that has never had a value set raises AttributeError when you try to access it).

(Each slot attribute costs you a pointer, either 4 or 8 bytes depending on your platform. An instance dictionary costs you at least a pointer plus sys.getsizeof(dict()) bytes. On my 32-bit and 64-bit Linux machines, this works out to about 35 or 36 attributes taking as much memory as an empty instance dictionary.)

Third, if you have a subset of instance attributes that are always present plus some unpredictable attributes that are added dynamically, you can potentially save space overall by preallocating space for the always-present attributes while still having an instance dictionary for the dynamic attributes. To do this you use a __slots__ with your attribute names plus "__dict__". This is the least straightforward and most conditional memory savings.

In the abstract, a slot uses about a third of the memory of a dictionary entry. In practice, well, dictionary space usage is complicated so how much memory this will save depends on the typical number of entries in the instance dictionary and how many entries you're able to preallocate. In current versions of CPython 2.x, the breakpoints you're likely to hit are 5 entries and 21 entries. Moving from six entries to five will save you a chunk of memory, but moving from five to four won't. In particular, an instance dictionary with no entries takes up just as much memory as one with five entries.

(Yes, this means that adding slots while keeping an instance dictionary can actually cost you more memory. That's why you really want to measure this sort of stuff.)

PS: I'm ignoring __weakref__ in this discussion. If you need weak references, you probably already know it and you can add it to your __slots__.

python/WhatSlotsAreGoodFor written at 01:56:23; Add Comment

Page tools: See As Normal.
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.