Wandering Thoughts archives

2013-11-17

The 10G Ethernet performance problem on Linux

It's clear that 10G Ethernet on Linux is not yet in the state that 1G Ethernet is, where you can simply assume that you'll get wire speed unless your hardware is terrible (but sometimes your hardware is terrible, or at least not great). Instead you need to tune things for best performance, and do so beyond the basics of MTU 9000 and large application buffers. There are even any number of resources on the web to tell you things about this; for example, I've recently been reading this one [PDF].

(There's also this one from 2008 [PDF slides] that I've seen referred to in a number of places.)

The problem here is simple: that paper is from 2009. Things have changed since 2009; in fact, I've seen things change between kernel 3.11.6 and kernel 3.12 (and they changed significantly between Ubuntu 12.04's 3.2.0 kernel and 3.11.6). Much of the other 10G tuning advice on the web I've found is like this, either clearly old or undated but probably old. Since they're old, some but not all of their performance tuning advice is likely out of date and either not necessary, not applicable any more, or actively counterproductive. Given the changes I've seen just between 3.11.6 and 3.12, this is probably going to continue to be the case for a while more; even carefully researched tuning advice written today may not apply in a year.

(At least not to current kernels. If you research tuning advice for, say, a RHEL/CentOS 6 kernel it's likely to stay useful for years because RHEL kernels don't change much.)

This is the 10G Ethernet performance problem on Linux as I see it. Today and for the likely future, getting good performance out of 10G Ethernet on Linux is going to take you real work. It's not enough to read some resources and follow their advice because parts of the advice may be out of date; you're going to have to experiment, ideally under real life scenarios not just artificial bandwidth or latency tests.

(Artificial tests can at best verify that under ideal circumstances you can hit wire bandwidth or wire latency. But the tuning you need for them may be different than the tuning you need for your live production load.)

linux/10GPerformanceProblem written at 20:44:50; Add Comment

Sending and receiving file descriptors in Python

On some but not all modern Unix systems, file descriptors (the underlying operating system level thing behind open files, sockets, and so on) can be passed between cooperating processes using Unix domain sockets and special options to sendmsg() and recvmsg(). There are a number of uses for this under various circumstances; the one that I'm interested in is selectively offloading incoming network connections from one process to another one that is better suited to handle some particular connections.

In Python 3.3 and later, doing this is simple because it is directly supported by the socket module. The documentation even includes code examples for both sendmsg() and recvmsg(), which is handy because they don't exactly have the most Pythonic of interfaces; instead it's basically a thin cover over the system call data structures. If you are receiving file descriptors that are sockets you're still left with the socket .fromfd() problem.

(I was encouraged to report the socket fd problem as an actual Python bug, where it has quietly been neglected just as I expected.)

Unfortunately Python 2 does not have any support for this in the socket module, thereby creating yet another gratuitous Python 2 to Python 3 difference. Fortunately a number of people have written add on modules to support this; the ones I found in a casual Internet search are python-passfd, python-fdsend, and sendmsg (which is notably lacking in documentation). Of these, python-fdsend seems to have the best API (and is packaged for Debian and Ubuntu); I expect that it's what I'll use if (or when) I need this feature in my Python 2 code. Note that it doesn't solve the socket .fromfd() problem.

If you're sending sockets to another process, remember that it is safe to call .close() on them afterwards but it is not safe to call .shutdown() on them; as I discovered, shutdown() is a global operation on a socket and applies to all file descriptors for it, including ones now held by other processes.

python/SendingFileDescriptors written at 02:01:49; Add Comment


Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.