What Python's global interpreter lock does (and doesn't) protect

November 28, 2005

Most people doing threading in Python know about Python's Global Interpreter Lock (GIL), which causes only one thread to be running in the CPython interpreter at any one time. This means that threaded code already has a certain amount of implicit locking going on, making certain operations thread-atomic without you writing explicit locks.

The important thing about the GIL for this is that it protects bytecodes, not Python statements. If something happens in a single bytecode, it's protected; otherwise, you need explicit locking. The most important thing done in single bytecodes is calls to methods implemented in C, such as all of the methods on builtin types. So things like list .append() or .pop() are atomic.

(All bets are off if you're dealing with someone's Python subclass of a builtin type, since it depends on what exactly they overrode.)

An important note is that in-place operations like '+=' are not actually single bytecodes, so counters like 'self.active += 1' need locks.

(Because they must be able to work on immutable objects, the actual INPLACE_* bytecodes result in an object, which is then stored back to the left hand side in a separate instruction. In-place operations on mutable builtin types can be atomic, but there aren't many mutable builtin types that support '+' et al.)

By default, Python only switches between runnable threads every hundred bytecodes or so, which can disguise code that isn't threadsafe. You can make Python switch almost every bytecode with sys.setcheckinterval(); see the sys module documentation. (Python opcodes don't always allow thread switching after themselves, so it's never literally every bytecode.)

(Again we see that implementation details matter, although there is an argument that this is too much black magic.)

Written on 28 November 2005.
« Weekly spam summary on November 26th, 2005
How not to set up your DNS (part 3) »

Page tools: View Source, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Nov 28 01:44:10 2005
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.