Wandering Thoughts archives

2017-05-24

Exploiting Python's Global Interpreter Lock for atomic operations is fun

Yesterday I wrote a sober and respectable article on how using the GIL safely has various tricky traps and you probably shouldn't try to do it. This is indeed the sensible, pragmatic approach to exploiting what the GIL protects for threaded Python code; you probably don't need that much performance and so you should use some form of explicit locking instead of trying to be clever.

The honest truth is that I've exploited the GIL this way before and I'll probably write more Python code that does it in the future. This isn't because it's necessarily a good idea; it's because the GIL is fun. Like many other complicated parts of CPython, the GIL is simply neat all by itself (at least if you're the kind of person who likes to peer behind the curtain and know how things work). And in general it's an important part of threaded CPython, with a lot of complexity that is important to understand if you want to write well-performing threaded Python code. You can't avoid learning something about the GIL's technicalities if you care about this area.

Once you have learned enough about the GIL, you have a Turing tarpit style puzzle of figuring out how to map your problem onto Python operations that are what I'll call GIL atomic, things that will never have races because the GIL is held across them. This sort of thing is like catnip for me and many programmers, especially since it requires deep domain knowledge. It make us feel smart, we're solving tough problems creatively and with real benefits (after all, this is the high performance option), and it means all of our domain knowledge gets put to nominally good use. It's simply fun to make CPython do what I want here, without the overhead and typing and work of setting up mutexes or managing locks or similar measures. It's magic; you do ordinary operations and things just work because you designed everything right.

(Then I can make myself feel even more clever by writing a long comment about how and why what I'm doing is perfectly safe.)

So that's my awkward confession about exploiting the GIL. It would be completely sensible to avoid doing so, but I'm probably not going to. Either I'll justify it on the grounds that this is a personal project and I'm allowed to have fun, or I'll justify it on the grounds of performance, but the truth is that I want to do it because it's neat and fun.

(If it was boring and tedious and took more work to arrange to exploit the GIL this way, you can bet that I'd be using mutexes and locks all the time.)

python/GILExploitingIsFun written at 00:59:43; Add Comment


Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.