Sending and receiving file descriptors in Python

November 17, 2013

On some but not all modern Unix systems, file descriptors (the underlying operating system level thing behind open files, sockets, and so on) can be passed between cooperating processes using Unix domain sockets and special options to sendmsg() and recvmsg(). There are a number of uses for this under various circumstances; the one that I'm interested in is selectively offloading incoming network connections from one process to another one that is better suited to handle some particular connections.

In Python 3.3 and later, doing this is simple because it is directly supported by the socket module. The documentation even includes code examples for both sendmsg() and recvmsg(), which is handy because they don't exactly have the most Pythonic of interfaces; instead it's basically a thin cover over the system call data structures. If you are receiving file descriptors that are sockets you're still left with the socket .fromfd() problem.

(I was encouraged to report the socket fd problem as an actual Python bug, where it has quietly been neglected just as I expected.)

Unfortunately Python 2 does not have any support for this in the socket module, thereby creating yet another gratuitous Python 2 to Python 3 difference. Fortunately a number of people have written add on modules to support this; the ones I found in a casual Internet search are python-passfd, python-fdsend, and sendmsg (which is notably lacking in documentation). Of these, python-fdsend seems to have the best API (and is packaged for Debian and Ubuntu); I expect that it's what I'll use if (or when) I need this feature in my Python 2 code. Note that it doesn't solve the socket .fromfd() problem.

If you're sending sockets to another process, remember that it is safe to call .close() on them afterwards but it is not safe to call .shutdown() on them; as I discovered, shutdown() is a global operation on a socket and applies to all file descriptors for it, including ones now held by other processes.

Written on 17 November 2013.
« Unix getopt versus Google's getopt variant and why Unix getopt wins
The 10G Ethernet performance problem on Linux »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun Nov 17 02:01:49 2013
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.