Wandering Thoughts archives

2022-02-21

Python's Global Interpreter Lock is not there for Python programmers

I recently read Evan Ovadia's Data Races in Python, Despite the Global Interpreter Lock (via), which discusses what its title says. Famously, the Global Interpreter Lock only covers the execution of individual Python bytecodes (more or less), and what this does and doesn't cover is tricky, subtle, and depends on the implementation details of Python code. For example, making a Python class better and more complete can reduce what's safe to do with it without explicit locking.

These days, I've come to feel that the Global Interpreter Lock is not really for Python programmers. Who the GIL is for is the authors of CPython packages that are written in C (or in general any compiled language). The GIL broadly allows authors of those packages to not implement any sort of locking in their own code, even when they're manipulating C level data structures, because they're guaranteed that their code will never be called concurrently or in parallel. This extends to the Python standard objects themselves, so that (in theory) Python dicts don't need any sort of internal locks in order to avoid your CPython process dumping core or otherwise malfunctioning spectacularly. Concurrency only enters into your CPython extension if you explicitly release the GIL, and the rules of the CPython API make you re-take the GIL before doing much with interpreter state.

(There are probably traps lurking even for C level extensions that allow calls back into Python code to do things like get attributes. Python code can come into the picture in all sorts of places. But for simple operations, you have a chance.)

Avoiding internal locks while looking into or manipulating objects matters a lot for single threaded performance (Python code looks into objects and updates object reference counts quite frequently). It also makes the life of C extensions simpler. I'm not sure when threading was added to Python (it was a very long time ago), but there might have been C extensions that predated it and which would have been broken in multi-threaded programs if CPython added a requirement for internal locking in C-level code.

The Global Interpreter Lock can be exploited by Python programmers; doing so is even fun. But we really shouldn't do it, because it's not designed for us and it doesn't necessarily work that well when we try to use it anyway. Python has a variety of explicit locking available in the standard library threading module, and we should generally use them even if it's a bit more annoying.

(Honesty compels me to admit that I will probably never bother to use locking around 'simple' operations like appending to a list or adding an entry to a dict. I suspect at least some people would even see using explicit locks for that (in threaded code) to be un-Pythonic.)

GILWhoItIsFor written at 22:28:58; Add Comment


Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.