Safely using Python's Global Interpreter Lock is quite tricky and subtle
Recently I ran across Grok the GIL: How to write fast and thread-safe Python (via), by A. Jesse Jiryu Davis. In the original version of the article, A. Jesse wrote:
Even though the line
lst.sort()takes several steps, the
sortcall itself is a single bytecode, and thus there is no opportunity for the thread to have the GIL seized from it during the call. We could conclude that we don't need to lock around
In comments on that article, Ben Darnell pointed out that this is not necessarily true. The fundamental issue is that Python objects are allowed to customize how they are compared for sorting, hashed for things like dictionary insertions, and so on. These operations necessarily require running Python code, and the moment you start running Python code the GIL may be seized from you.
The version of the article on A. Jesse's personal site has been updated to note some qualifications:
[...] so long as we haven’t provided a Python callback for the
keyparameter, or added any objects with custom
If you guessed that these qualifications are not quite complete,
you would be correct; they're missing the rich comparison operations,
which may be defined instead of
__cmp__ (and in Python 3, they're
the only way to customize comparisons and ordering).
This may seem like nit-picking and in a sense it is. But what I
really want to illustrate here is just how tricky it is to safely
use the GIL without locking. In CPython there is a huge number of
special cases where Python code can start running in what appear
to be simple atomic C-level operations like
these special cases requires a great deal of up to date Python
knowledge (so you know all the various ways things can be customized),
then finding out if they're going to affect any particular operation
you want to do may require deep dives into the source code you're
dealing with. This is especially the case if the code uses things
like metaclasses or class decorators.
In practice things get worse, because what you've looked at is a
single point in time of your codebase; you only know that it's safe
today. Nothing fundamentally prevents someone coming along later
to helpfully add some rich comparison operators to a class that
invalidate your assumptions that
lst.sort() will be an atomic
operation as far as the GIL is concerned and so you don't need to
do any locking around it.
(If everyone involved is lucky, the class comments document that other code depends on the class not having rich comparison operators or whatever else you need it to not have.)
So what I've wound up feeling is that GIL safety is generally too complicated and tricky to use. Or perhaps it would be better to say that it's too fragile, since there are a vast number of ways to accidentally destroy it without realizing what you've done. If you actively want to use GIL safety to avoid explicit locking, it's probably going to be one of the trickier portions of your code and you should be very careful to document everything about it and to keep it as simple as possible (for example, using only primitive C-level types even if this requires contortions).
It's unfortunate that the GIL is this way, but it is (at least for now in CPython and thus probably for the future).
(In theory CPython could be augmented so that operations like
lst.sort() explicitly held the GIL for their entire duration so
that people wouldn't get surprised this way. But I suspect that the
CPython developers want people to use explicit locking, mutexes,
and so on, and not rely on hard to explain GIL guarantees that
constrain their implementation choices.)
What I mostly care about for speeding up our Python programs
There are any number of efforts and technologies around these days that try to speed up Python, starting with the obvious PyPy and going on to things like Cython and grumpy. Every so often I think about trying to apply one of them to the Python code I deal with, and after doing this a few times (and even making some old experiments with PyPy) I've come to a view of what's important to me in this area.
What I've come around to caring about most is reducing the overall resource usage of short running programs that mostly use the Python standard library and additional pure-Python modules. By 'resource usage' I mean a combination of both CPU usage and memory usage; in our situation it's not exactly great if I make a program run twice as fast but use four times as much memory. In fact for some programs I probably care more about memory usage than CPU, because in practice our Python-based milter system probably spends most of its time waiting for our commercial anti-spam system to digest the email message and give it a verdict.
(Meanwhile, our attachment logger program is probably very close to being CPU bound. Yes, it has to read things off disk, but in most cases those files have just been written to disk so they're going to be in the OS's disk cache.)
I'm also interested in making DWiki (the code behind Wandering Thoughts) faster, but again I actually want it to be less resource-intensive on the systems it runs on, which includes its memory usage too. And while DWiki can run in a somewhat long-running mode, most of the time it runs as a short-lived CGI that just serves a single request. DWiki's long-running daemon mode also has some features that might make it play badly with PyPy, for example that it's a preforking network server and thus that PyPy is probably going to wind up doing a lot of duplicate JIT translation.
I think that all of this biases me towards up-front approaches like Cython and grumpy over on the fly ones such as PyPy. Up-front translation is probably going to work better for short running programs (partly because I pay the translation overhead only once, and in advance), and the results are at least reasonably testable; I can build a translated version and see in advance whether the result is probably worth it. I think this is a pity because PyPy is likely to be both the easiest to use and the most powerful accelerator, but it's not really aimed at my usage case.
(PyPy's choice here is perfectly sensible; bigger, long-running programs that are actively CPU intensive for significant periods of time are where there's the most payoff for speeding things up.)
PS: With all of this said, if I was serious here I would build the latest version of PyPy by hand and actually test it. My last look and the views I formed back then were enough years ago that I'm sure PyPy has changed significantly since then.
Why modules raising core exceptions mostly hurts, not helps, your users
A while back I wrote an entry about how modules should never raise core Python exceptions. Recently via my Referer logs I found out that some people aren't convinced by my entry, so I feel like taking another run at this topic, this time approaching it from the perspective of someone using your module.
If I'm invoking some function or method from your module and want to trap errors, I need to write code like this:
import yourmoddef fred(): try: res = yourmod.some_thing(10, 20) except SOMETHING as e: ...
In order to fill in that
SOMETHING with the right exception, I
need to consult your module's documentation. Given that I have to
look this up, reusing a general exception saves me essentially
nothing; at most I type a little less, and
RuntimeError is not exactly a compelling savings. If I just want
to catch your explicitly raised errors, using
RuntimeError (or a
subclass of it) is not saving me any real effort.
In practice, only catching explicitly raised errors is almost always what people using your module want to do, because of the various dangers of over-broad tries that I mentioned in my original entry and elsewhere. And if I really do want to catch all errors that come out of your code, I can already do that explicitly:
try: res = yourmod.some_thing(10, 20) except Exception as e: ...
Notice that raising
RuntimeError instead of your own error doesn't
actually help me here. If I want to catch all possible errors that
can happen during your module's execution, I need to go much broader
(There are valid cases for doing this broad catching, generally in top-level code that wants to insure that no uncaught exceptions ever surface to the user.)
Which brings me around to the one case where it is sensible to raise
standard errors, which is when you're writing code that stands
in for standard Python code that raises these errors. This is the
one case where using a standard error saves me from looking things
up; in fact, using a standard error is generally essential. If
you're writing a class that will get used instead of a standard
KeyError and so on is absolutely essential,
because that makes your objects transparent substitutes for real
(This is in a sense a matter of API compatibility, where the
exceptions that get raised are part of the API. Sometimes this can
be explicit, as in the case of
KeyError, but sometimes this is
more implicit, in cases where raising and catching errors is more
Sidebar: my view on subclassing
I don't think you should do this because I don't think it's useful.
If someone is catching
RuntimeError today they probably intend
to catch significant internal issues inside Python, not errors that
your module happens to raise. Sweeping your module's errors into
except clauses is probably not helpful and may even be
For better or worse, I think that there is (or should be) a strong separation between 'internal' Python errors raised by the CPython interpreter and core Python modules, and module-specific errors raised by modules (yours, third party modules, and non-core modules in the standard library). These two sorts of errors don't really live in the same taxonomy and so putting one under the other is generally not going to help anyone.
Part of why Python 3.5's
async have some odd usage restrictions
Python 3.5 added a new system for coroutines and asynchronous
programming, based around new
await keywords (which
have the technical details written up at length in PEP 492). Roughly speaking, in
terms of coroutines implemented with
await replaces '
yield from' (and is
more powerful). So what's
async for? Well, it marks a function
that can use
await. If you use
await outside an
you'll get a syntax error. Functions marked
async have some odd
restrictions, too, such as that you can't use
from in them.
When I described doing coroutines with
yield from here, I noted that it was potentially error
prone because in order to make everything work you had to have an
unbroken chain of
yield from from top to bottom. Break the chain
yield instead of
yield from, and things wouldn't work.
And because both
yield from and
yield are used for regular
generators as well as coroutines, it's possible to slip up in
various ways. Well, when you introduce new syntax you can fix
issues like that, and that's part of why
have their odd rules.
A function marked
async is a (native) coroutine.
await can only
be applied to coroutines, which means that you can't accidentally
treat a generator like a coroutine the way you can with
from. Simplifying slightly, coroutines can only be invoked through
await; you can't call one or use them as a generator, for example
for something in coroutine(...):'. As part of not being
generators, coroutines can't use '
yield' or '
(And there's only
await, so you avoid the whole '
yield from' confusion.)
In other words, coroutines can only be invoked from coroutines and
they must be invoked using the exact mechanism that makes coroutines
work (and that mechanism isn't and can't be used for or by anything
else). The entire system is designed so that you're more or less
forced to create that unbroken chain of
awaits that makes it all
go. Although Python itself won't error out on
import time if you
try to call a
async function without
await (it just won't work
at runtime), there's probably Python static checkers that look for
this. And in general it's an easy rule to keep track of; if it's
async, you have to
await it, and this status is marked right
there in the function definition.
(Unfortunately it's not in the type of the function, which means
that you can't tell by just importing the module interactively and
then doing '
Sidebar: The other reason you can only use
Before Python 3.5, the following was completely valid code:
def somefunc(a1, b2): ... await = interval(a1, 10) otherfunc(b2, await) ...
In other words,
await was not a reserved keyword and so could be
legally used as the name of a local variable, or for that matter a
function argument or a global.
Had Python 3.5 made
await a keyword in all contexts, all such
code would immediately have broken. That's not acceptable for a
minor release, so Python needed some sort of workaround. So it's
not that you can't use
await outside of functions marked
it's that it's not a keyword outside of async functions. Since
it's not a keyword, writing something like '
await func(arg)' is
a syntax error, just as '
abcdef func(arg)' would be.
The same is true of
async, by the way:
def somefunc(a, b, async = False): if b == 10: async = True ....
Thus why it's a syntax error to use '
async for' or '
outside of an
async function; outside of such functions
isn't even a keyword so '
async for' is treated the same as '
(I'm sure this makes Python's parser that much more fun.)
How we can use
yield from to implement coroutines
Give my new understanding of generator functions and
yield from, we can now see how to use
from to implement coroutines and an event loop. Consider a three
level stack of functions, where on the top layer you have an event
loop, in the middle you have the processing code you write, and on
the bottom are event functions like
Let's start with an example processing function or two:
def countdown(n): while n: print("T-minus", n) n -= 1 yield from sleep(1) def launch(what, nsecs): print("counting down for", what) yield from countdown(nsecs) print("launching", what)
To start a launch, we call something like '
10))', which looks a bit peculiar since it sort of seems like
coro.start() should get control only after the launch. However,
we already know that calling a generator function doesn't do exactly
what it looks like. What
coro.start() gets when we do this is an
unstarted generator object (which handily encapsulates those
launch(), so we don't have to do it by hand).
When the coroutine scheduler starts the
launch() generator object,
we wind up with a chain of
yield froms that bottoms out at
sleep() yields is passed back up to the coroutine
scheduler and the entire call chain is suspended; this is no different
that what I did by calling
.send() by hand yesterday. What
sleep() returns to the
scheduler is an object (call it an event object) that tells the
coroutine scheduler under what conditions this coroutine should be
resumed. When the scheduler reaches the point that the coroutine
should be run again, the scheduler will once again call
which will resume execution in
sleep(), which will then return
countdown(), and so on. The scheduler may use this
to pass information back to
sleep(), such as how long it took
before the coroutine was restarted.
yield from are being used for two things. First,
they create a communication channel between the coroutine scheduler
and the low-level event functions like
countdown() functions are oblivious to this since they don't
touch either the value
sleep() yields up to the scheduler or the
value that the scheduler injects to
the chain of
yield from and the final
yield neatly suspend the
entire call stack.
In order for this to work reliably, there are two rules that our
user-written processing functions have to follow. First, they must
never accidentally attempt to do anything with the
function. It is okay but unclear for a non-generator function to
sleep() and return the result:
def sleep_minutes(n): return sleep(n * 60) def long_countdown(n): while n: print("T-minus", n, "minutes") yield from sleep_minutes(1) n -= 1
This is ultimately because '
yield from func()' is equivalent to
t = func(); yield from t'. We don't care just how the generator
object got to us so we can
yield from it, we just care that it
However, at no stage in our processing functions can we attempt to
look at the results of iterating
sleep()'s generator object, either
directly or indirectly by writing, say, '
for i in countdown(10):'.
This rules out certain patterns for writing processing functions, for
instance this one:
def label_each_sec(label, n): for _ in tick_once_per_sec(n): print(label)
This leads to the second rule, which is that we must have an
unbroken chain of
yield froms from the top to the bottom of our
processing functions, right down to where you use an event function
sleep(). Each function must 'call' the next using the
yield from func()' idiom. In effect we don't have calls from one
processing function to another; instead we're passing control from
one function to the next. In my example,
launch() passes control
countdown() until the countdown expires (and
passes control to
sleep()). If we actually call a processing
function normally or accidentally use '
yield' instead of '
from', the entire collection explodes into various sorts of errors
without getting off the launch pad and you will not go to space
As you might imagine, this is a little bit open to errors. Under
normal circumstances you'll catch the errors fairly fast (when
your main code doesn't work). However, since errors can only be
caught at runtime when a non-
yield from code path is reached,
you may have mistakes that lurk in rarely executed code paths.
Perhaps you have a rarely invoked last moment launch abort:
def launch(what, nsecs): print("counting down for", what) yield from countdown(nsecs) if launch_abort: print("Aborting launch! Clear the launch pad for", what) yield sleep(1) print("Flooding fire suppression ...") else: print("launching", what)
It might be a while before you discovered that mistake (I'm doing
a certain amount of hand-waving about early aborts in
(See also my somewhat related attempt at understanding this sort
help asynchronous programming.
Note that you can't use my particular approach from that entry in
Python with '
yield from' for reasons beyond the scope of this
Sorting out Python generator functions and
yield from in my head
Through a chain of reading, I wound up at How the heck does
async/await work in Python 3.5? (via
the Trio tutorial).
As has happened before when I started reading about Python 3's new
await stuff, my head started hurting when I hit the
somewhat breezy discussion of
yield from and I felt the need to
slow down and try to solidly understand this, which I haven't really
Generator functions are functions that contain a
def fred(a): r = yield 10 print("fred got:", r) yield a
A straightforward generator function is there to produce a whole series of values without having to ever materialize all of them at once in a list or the like.
Calling a generator function does not return its result. Instead, it returns a generator object, which is a form of iterator:
>>> fred(10) <generator object fred at 0x7f52a75ea1a8>
This generator object is in part a closure that captures the argument
fred() was called with (and in general will preserve
state while it is repeatedly iterated). Note that
doesn't start executing until you try to get the first value from
One common pattern with a stack of generator functions (including
needing to modify or filter part of a generator's results) was that you would have one generator
function that wanted to call another one for a while. In the beginning
this was done with explicit
for loops and the like, but then
yield from takes a generator or iterator
and exhausts it for you, repeatedly
yield'ing the result.
def barney(a): yield from fred(a)
(You can intermix
yield from and use both of them
more than once in a function.)
Because generator functions actually return generators, not any
sort of result, '
yield from func()' is essentially syntactic sugar
for calling the function, getting a generator object back, and then
yield from on the generator object. There is no special
magic involved in that:
def barney(a): gen = fred(a) yield from gen
Because generator objects are ordinary objects, they can be returned through functions that are not generators themselves, provided that intermediate functions don't really attempt to manipulate them and simply return them as-is:
def jim(a): return fred(a) def bob(a): yield from jim(a)
jim() actually iterated through
fred()'s results, things
would quietly go sideways in ways that might or might not be visible.)
yield started out, it was a statement; however, that got
revised so that it was an expression and could thus have a value,
as we see in
fred() where the value of one
yield is assigned
r and then used later.
You (the holder of the generator object) inject that value by
.send() on the generator:
>>> g = fred(2) >>> _ = g.send(None); _ = g.send("a") fred got: a
.send() starts the generator running and must be made
As part of adding
yield from, Python arranged it so that if you
had a stack of
yield from invocations and you called
on the outer generator object, the value you sent did not go to
the outer generator object; instead it goes all the way down to
the eventual generator object that is doing a
yield instead of
def level3(a): # three levels of yield from # and we pass through a normal # functions too yield from bob(a) >>> g = level3(10) >>> _ = g.send(None); _ = g.send("down there") fred got: down there
This means that if you have a stack of functions that all relay
things back up using '
yield from', you have a direct path from
your top level code (here that's our interactive code where we
level3()) all the way down to the core generator function
at the bottom of the call stack (here, the
fred() function). You
and it can communicate with each other through the values it
and the values you
send() to it without any function in the middle
having to understand anything about this; it's entirely transparent
(Don't accidentally write '
yield' instead of '
though. The good news about that mistake is that you'll catch
Hopefully writing this has anchored
yield from's full behavior
and the logic behind it sufficiently solidly in my head that it
will actually stick this time around.
yield from versus
yield of a generator
Suppose that we have a little mistake:
def barney2(a): yield fred(a)
What happens? Basically what you'd expect:
>>> list(barney(20)) fred got: None [10, 20] >>> list(barney2(20)) [<generator object fred at 0x7f0fdf4b9258>]
When we used
yield instead of
yield from, we returned a value
instead of iterating through the generator. The value here is what
we get as the result of calling
fred(), which is a generator
By the way, a corollary to strings being iterable is that accidentally calling '
from' instead of '
yield' on a string won't fail the way that eg
yield from 10' does but will instead give you a sequence of
single characters. You'll probably notice that error fairly fast,
This behavior of
yield from is pretty much a feature, because it
means that you can
yield from another function without having to
care about whether it's an actual generator function or it merely
returns an iterable object of some sort; either will work.
In Python, strings are infinitely recursively iterable
Yesterday I posed the side question of what
happened when you called the following code with
def flatten2(inlst): olst =  for i in inlst: try: it = iter(i) except TypeError: it = None if it is None: olst.append(i) else: olst.extend(flatten2(i)) return olst
The intent of this code is to recursively flatten iterable things.
What I had expected to get when I called it with a string was
'b', 'c', 'd', 'e', 'f'], ie that it flattened the string instead
of leaving it alone (because strings are iterable). What I actually
got (and what you can get) is a RecursionError of 'maximum recursion
depth exceeded'. So why did this recurse endlessly?
The answer is that strings are not just iterable, they are infinitely
recursively iterable. With normal iterable containers, iterating
the container yields non-iterable items unless you've explicitly
put in iterable ones (such as a list inside another list); assuming
that you have not cleverly embedded a cycle in your container, our
recursive flattening will eventually bottom out and finish. This
is not the case for Python strings. When you iterate a multi-character
"abcdef", you get a sequence of single-character
"a" "b" "c" "d" "e" "f". However, these are still strings
and so they are still iterable; when you iterate the
you get back another single-character string
"a", which is also
iterable. And so
flatten2() chases down an infinite recursion of
trying to unpack single-character strings into non-iterable items,
which it will never succeed at.
Fixing this without side effects is a bit annoying. It's not enough
to immediately check that
inlst is length 1 and just return it
if so, because this fails on
flatten2(["abcdef",]) too, for that matter). I think you have to
check explicitly for length 1 strings (and bytes) and just return
them, which of course leaves you exposed to any other infinitely
recursively iterable types (should there be any other out there).
(I was about to gripe about
repr()'s representation of single
character strings but then I tested and it uses single quotes for
all strings, not just single-character ones. I need to remember
that in Python there is no 'character' type that's different from
strings, unlike in other languages such as Go, and so single quotes
still mean 'string'.)
Sidebar: It's trivially possible to create containers with cycles
>>> a = [1, 2, 3] >>> a.append(a) >>> print(a) [1, 2, 3, [...]]
More involved versions are left as an exercise for the reader.
Without contemplating it too hard, I think that you can only create cycles with mutable containers, and then only with some of them. Sets are mutable, for example, but they can only contain hashable items, which are generally immutable.
How recursively flattening a list raises a Python type question
Today I wound up reading Why it's hard for programmers to write
a program to flatten a list?
where the quiz challenge put forward is to turn an input like
[1,[2,3], [4, [5,6]]] into
[1,2,3,4,5,6]. My immediate reaction
was that I'd do this in Python rather than in any statically typed
language I know, because all of them make the input type here hard
to represent. But then I realized that doing this in Python raises
another type-related question.
If we stick exactly to the specification (and directly implement it), the result is pretty simple and straightforward:
def flatten(inlst): olst =  for i in inlst: if isinstance(i, int): olst.append(i) elif isinstance(i, list): olst.extend(flatten(i)) else: raise ValueError("invalid element in list") return olst
(You can optimize this by having a
_flatten internal function
that gets passed the output list, so you don't have to keep building
lists and then merging them into other lists as you work down and
then back up the recursion stack. Also, I'm explicitly opting to
return an actual list instead of making this a (recursive) generator.)
However, this code is not very Pythonic because it is so very
narrowly typed. We can relax it slightly by checking for
(int, float)), but even then most people would say that
should definitely accept tuples in place of lists and probably even
If we're thinking about being Pythonic and general, the obvious thing to do is check if the object is iterable. So we write some simple and general code:
def flatten2(inlst): olst =  for i in inlst: try: it = iter(i) except TypeError: it = None if it is None: olst.append(i) else: olst.extend(flatten2(i)) return olst
This should flatten any type (or mixture of types) that contains
elements, as denoted by the fact that it's iterable. It looks good
and passes initial tests.
Then some joker calls our code with
suddenly we have a problem. Then another joker calls our code
flatten2([somedict,]) and files a bug that our code only
flattens the keys of their dictionary, not the keys and values.
(As an exercise, can you predict in advance, without trying it,
what our problem is with
flatten2(["abcdef",]), and why it
happens? I got this wrong when I was writing and testing this
code in Python 3 and had to scratch my head for a bit before the
The problem here is that 'is iterable' is not exactly what we want.
Some things, such as strings, are iterable but should probably be
treated as indivisible by
flatten2(). Other things, such as dicts,
are iterable but the default iteration result does not fully represent
their contents. Really, not only is Python lacking a simple condition
for what we want, it's arguably not clear just what we want to do
if we're generalizing
flatten() (and what making it 'Pythonic'
One valid answer is that we will explicitly check for container
types that are close enough to what we want, and otherwise mostly
return things as-is. Here we would write a version of
that looked like this:
def flatten3(inlst): olst =  for i in inlst: if isinstance(i, (list, tuple, set)): olst.extend(flatten3(i)) elif isinstance(i, dict): raise ValueError("dict not valid in list") else: olst.append(i) return olst
We could treat dicts as single elements and just return them, but that is probably not what the caller intended. Still, this check feels dubious, which is a warning sign.
As a minimum, it would be nice to have a Python abstract type or trait that represented 'this is a container object and iterating it returns a full copy of its contents'; you could call this the property of being list-like. This would be true for lists, tuples, and sets, but false for dicts, which would give us a starting point. It would also be true for strings, but you can't win them all; when dealing with iterable things, we'll probably always have to special-case strings.
(I'd go so far as arguing that making strings iterable by default was a Python mistake. It's one of those neat features that winds up getting in the way in practice.)
I don't have an answer here, by the way. If I was in this situation
I might either write and carefully document a version of
(specifying 'recursively flattens any iterable thing using its
default iterator; this will probably not do what you want for
dicts'), or go with some version of
flatten3() that specifically
restricted iteration to things that I felt were sufficiently
(I'd worry about missing some new popular type over time, though.
Ten years ago I might not have put
set in the list, and who knows
what I'm missing today that's going to be popular in Python in the
future. Queues? Trees? Efficient numerical arrays?)
Python won't (and can't) import native modules from zip archives
I've written before about running Python programs from zip files, and in general you can package up Python modules in zip files. Recently I was grumbling on Twitter about the hassles of copying multi-file Python things around, and Jouni Seppänen mentioned using zip files for this but wasn't sure whether they supported native modules (that is, compiled code, as opposed to Python code). It turns out that the answer is straightforward and unambiguous; Python doesn't support importing native modules from zip files. This is covered in the zipimport module:
Any files may be present in the ZIP archive, but only files
.pycare available for import. ZIP import of dynamic modules (
.so) is disallowed. [...]
(The Python 2.7 documentation says the same thing. If you're like
me and had never heard of
.pyd files before, they're basically
I don't know about the Windows side of things, but on Unix (with
.so shared objects), this is not an arbitrary restriction Python
has imposed, which is what the term 'disallowed' might lead you to
think. Instead it's more or less inherent in the underlying API
that Python is using. Python loads native modules using the Unix
dlopen() function, and the
dlopen() manpage is specific
about its API:
void *dlopen(const char *filename, int flags);
Which is to say that
dlopen() takes a (Unix) filename as the
dynamic object to load. In order to call
dlopen(), you must have
an actual file on disk (or at least on something that can be
mmap()'d). You can't just hand
dlopen() a chunk of memory, for
example a file that you read out of a zip file on the fly, and say
'treat this as a shared object'.
dlopen() relies on
mmap() for relatively solid reasons due to
how regular shared libraries are loaded in order to share memory
between processes. Plus, wanting to turn a
block of memory into a loaded shared library is a pretty uncommon
thing; mapping existing files on disk is the common case. There are
probably potential users other than Python, but I doubt there are
In theory perhaps Python could extract the
.so file from the zip
file, write it to disk in some defined temporary location, and
dlopen() that scratch file. There are any number of potential
issues with that (including fun security ones), but what's certain
is that it would be a lot of work. Python declines to do that work
for you and to handle all of the possible special cases and so on;
if you need to deploy a native module in a bundle like this, you'll
have to arrange to extract it yourself. Among other things, that
puts the responsibility for all of the special cases on your
shoulders, not on Python's.
I think docstrings in Python are for everything, not just public things
In my entry on my mixed feelings about docstrings, I mentioned that my most recent Python code was all comments and no docstrings, but also that it was a program instead of a module. So here's a question: should that matter for whether or not you use docstrings? The obvious related question is whether docstrings are for documenting everything in a module, or only the things people are supposed to use.
My best current answer is that if you're going to use docstrings at
all, I think that they should be for everything. It's not that
docstrings are how you make public documentation, things that you
intend that people will use
help() on and so on; instead, it's
that docstrings are simply the Pythonic way to write function,
method, class, and module documentation, whether those are intended
for public use or not. Signaling which things are public versus
private is best done through other mechanisms (such as a module
__all__ with public exports).
(At this point I will pause to admit that I took a quick look through the standard library and it doesn't seem to be entirely consistent on this. Most things that appear to be internal functions and that have documentation at all seem to use docstrings, but some of them have one or two line comments at the start where docstrings would normally go.)
Of course there are some things that you can't document with docstrings; constants and variables can't have docstrings attached, so you need to either use comments or cover them in the module level documentation.
(Here I'm talking about 'what this function does and how you call it' sort of documentation. Other sorts of commentary about the code and how things operate still go in comments because there isn't any other place where they belong, so you'll find those sort of comments scattered through the standard library as necessary.)
I also have no idea if this is a widespread view in the Python
community. I have the impression that it is and the Django source
code seems to use docstrings to document functions and so on, but
it could be that people now see comments as the way to go outside
of public stuff that should show up in
(I'll probably try to shift how I document things to docstrings for the next piece of Python code I write, assuming I remember this by then and don't give up in annoyance.)