Wandering Thoughts archives

What happens when CPython deletes a module

The first thing to know about what happens when Python actually deletes and garbage-collects a module is that it doesn't happen very often; in fact, you usually have to go out of your way to observe it. The two ways I know of to force a module to be deleted and garbage collected are to end your program (which causes all modules to eventually go through garbage collection) and to specifically delete all references to a module, including in sys.modules. People don't do the latter very often so mostly this is going to come up at program shutdown.

(Before I started investigating I thought that reloading a module might destroy the old version of it. It turns out that this is not what happens.)

As I mentioned in passing a long time back, CPython actually destroys modules in a complex multi-pass way. First, all module level names that start with a single underscore are set to None. This happens in some random order (actually dictionary traversal order) and if this drops an object's reference count to nothing it will be immediately garbage collected. Second, all names except __builtins__ are set to None (again in the arbitrary dictionary traversal order).

Note that object garbage collections are not deferred until all entries have been set to None; they happen immediately, on the fly. If you peek during this process, for example in a __del__ method on an object being cleaned up, you can see some or all of the module-level variables set to None. Which ones in your pass are set to None is random and likely variable.

There are two comments in the source code to sort of explain this. The first one says:

To make the execution order of destructors for global objects a bit more predictable, we first zap all objects whose name starts with a single underscore, before we clear the entire dictionary. We zap them by replacing them with None, rather than deleting them from the dictionary, to avoid rehashing the dictionary (to some extent).

Minimizing the amount of pointless work that's done when modules are destroyed is important because it speeds up the process of exiting CPython. Not deleting names from module dictionaries avoids all sorts of shuffling and rearrangement that would otherwise be necessary, so is vaguely helpful here. Given that the odd two-pass 'destructor' behavior here is not really documented as far as I know, it's probably mostly intended for use in odd situations in the standard library.

The other comment is:

Note: we leave __builtins__ in place, so that destructors of non-global objects defined in this module can still use builtins, in particularly 'None'.

What happens to object destructors in general during the shutdown of CPython is a complicated subject because it depends very much on the order that modules are destroyed in. Having looked at the CPython code involved (and how it differs between Python 2 and Python 3), my opinion is that you don't want to have any important object destructors running at shutdown time. You especially don't want them to be running if they need to finalize some bit of external state (flushing output or whatever) because by the time they start running, the infrastructure they need to do their work may not actually exist any more.

If you are very curious you can watch the module destruction process with 'python -vv' (in both Python 2 and Python 3). Understanding the odd corners of the messages you see will require reading the CPython source code.

(This entire excursion into weirdness was inspired by a couple of tweets by @eevee.)

Sidebar: The easy way to experiment with this

Make yourself a module that has a class with a __del__ method that just prints out appropriate information, set up some module level variables with instances of this class, and then arrange to delete the module. With suitable variables and so on you can clearly watch this happen.

The easy way to delete your test module is:

import test
del test
del sys.modules["test"]

Under some circumstances you may need to force a garbage collection pass through the gc module.