Python won't (and can't) import native modules from zip archives

February 10, 2017

I've written before about running Python programs from zip files, and in general you can package up Python modules in zip files. Recently I was grumbling on Twitter about the hassles of copying multi-file Python things around, and Jouni Seppänen mentioned using zip files for this but wasn't sure whether they supported native modules (that is, compiled code, as opposed to Python code). It turns out that the answer is straightforward and unambiguous; Python doesn't support importing native modules from zip files. This is covered in the zipimport module:

Any files may be present in the ZIP archive, but only files .py and .pyc are available for import. ZIP import of dynamic modules (.pyd, .so) is disallowed. [...]

(The Python 2.7 documentation says the same thing. If you're like me and had never heard of .pyd files before, they're basically Windows DLLs.)

I don't know about the Windows side of things, but on Unix (with .so shared objects), this is not an arbitrary restriction Python has imposed, which is what the term 'disallowed' might lead you to think. Instead it's more or less inherent in the underlying API that Python is using. Python loads native modules using the Unix dlopen() function, and the dlopen() manpage is specific about its API:

void *dlopen(const char *filename, int flags);

Which is to say that dlopen() takes a (Unix) filename as the dynamic object to load. In order to call dlopen(), you must have an actual file on disk (or at least on something that can be mmap()'d). You can't just hand dlopen() a chunk of memory, for example a file that you read out of a zip file on the fly, and say 'treat this as a shared object'.

(dlopen() relies on mmap() for relatively solid reasons due to how regular shared libraries are loaded in order to share memory between processes. Plus, wanting to turn a block of memory into a loaded shared library is a pretty uncommon thing; mapping existing files on disk is the common case. There are probably potential users other than Python, but I doubt there are very many.)

In theory perhaps Python could extract the .so file from the zip file, write it to disk in some defined temporary location, and dlopen() that scratch file. There are any number of potential issues with that (including fun security ones), but what's certain is that it would be a lot of work. Python declines to do that work for you and to handle all of the possible special cases and so on; if you need to deploy a native module in a bundle like this, you'll have to arrange to extract it yourself. Among other things, that puts the responsibility for all of the special cases on your shoulders, not on Python's.

Comments on this page:

In Google there were (are?) .par files. They're self-contained python executables that can include native code. IIRC they extracted to a secure temp dir; this script is what makes them. Not sure if it's the most up to date version.

By David Sowder at 2017-02-11 19:42:44:

PyInstaller also optionally supports the single zip with unpack to temporary directory method of distribution.

Written on 10 February 2017.
« Malware strains may go away sometimes, but they generally come back
Different ways you can initialize a RAID-[567+] array »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Feb 10 01:40:40 2017
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.