What you can (probably) count on for concurrency in Python

July 28, 2008

A comment on the previous entry about how builtins are atomic asked a really good question:

Is there actually a guarantee in CPython that they're atomic, or is that a side-effect of implementation?

There are two answers, at least in my opinion. The legalistic answer is that there is no guarantee at all, because there's nothing in the documentation. But this is because the CPython documentation doesn't talk about concurrency issues at all; even the thread module documentation doesn't really say anything. In theory you can go from 'the documentation doesn't promise anything' to 'the documentation allows an implementation to do anything if you don't explicitly lock'. However, this is robot logic, not human logic; the real truth is that threading is not really a core Python feature (even today it is in the 'optional operating system' modules section), and the documentation reflects that.

I maintain that the practical answer is that it is guaranteed, at least for CPython, despite the lack of explicit documentation to that effect. Whether legalistic purists like it or not, in practice if you do not document something and then expose a single implementation to people, that implementation's long standing behavior becomes implicit documentation. This is especially so if the behavior is quite useful and the alternatives painful.

Or in short: it's guaranteed because there's no documentation about it either way and that's how CPython has always behaved. And it's really handy.

And as a pragmatic matter, anything that wants to be compatible with existing CPython threaded code is going to have to mimic this behavior (given Python 3K, this is less reassuring than it seems). However, IronPython, Jython, and any other similar projects out there may be not all that interested in supporting existing CPython threaded code; I suspect that they feel that they have better native threading models that you should use instead.

(Although for what it is worth IronPython does not document any differences in what is and isn't atomic for basic types in their list of differences between IronPython and CPython.)

(This is one of those entries that is going to get me pilloried.)

Comments on this page:

From at 2008-07-29 04:52:02:

This discussion semi-reminds me of another one going on at the moment where not many people will agree with the OP :). From Python-URL:

   A long thread against the "Explicit is better than implicit" principle:

-- aj

From at 2008-07-29 15:07:40:

It's a grey area where there is no foreseeable reason to change it, but should a reason be found, historical practices will not be sufficient argument against.

There are some things you can rely on, but they're not directly useful. Python always produces a sane exception, rather than a segfault - good for debugging, but doesn't help write the program.

Recursive access to a module, class, or instance dict is intentionally supported. Much of dict's atomicity is a side effect from this. The important part though is access to a distinct keys needs to be safe. Access to the same key is somewhat arbitrary.

You could try to extend that to concurrent access from threads, but keep in mind it's only module, class, and instance dicts - not arbitrary dicts of your own.

And the punchline.. use a python class as a key. The moment any python code is called for comparison, or even __del__ or a weakref callback, the operation stops being atomic.

-- Adam Olsen, aka Rhamphoryncus

By cks at 2008-07-30 01:35:57:

I disagree about the lack of weight given to historical (or current) practice, for reasons that are long enough that I put them in a new entry, PracticalLanguageGuarantees.

The issue of Python class instances as keys is a good point and an interesting question that is going to make me go read the CPython source code.

Written on 28 July 2008.
« Another advantage of Python builtins
What is guaranteed in languages in practice »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Jul 28 23:23:28 2008
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.