Van Jacobson illustrates the importance of cache effects

February 3, 2006

One of the most exciting presentation at the recent linux.conf.au 2006 conference (ironically held in New Zealand) was Van Jacobson's talk on speeding up the Linux networking stack. On a TCP stack that he says is already one of the fastest going, he managed to well over double the performance, while removing and simplifying code, even in driver hot paths.

(For more details, see David Miller's summary here; Van Jacobson's slides are here (1.4 MB PDF). Both are well worth reading.)

One of the things Van Jacobson did in this was to convert many of the queues used in the networking layers from linked lists (with locks) to what he calls a 'cache aware, cache-friendly queue' that is also lock-free. Converting just the driver level queues to these channels resulted in CPU usage dropping from 78% to 58% on a dual-CPU test machine.

This is not black magic; instead, this is a vivid illustration of just how much locks and cache contention cost you in practice. Multiple writers and atomic operations are now hugely expensive, so as Van Jacobson says, 'to go fast you want to have a single writer per [cache] line and no locks' (page 21 of his slides).

I can't help but note that that Van Jacobson's channels and his overall TCP processing architecture built around them don't look very much like the conventional threaded way to do parallel programming. Instead they smell a lot more like Hoare's CSP to me.


Written on 03 February 2006.
« The rise of wikiblogs
Why simple markup languages make sense »

These are my WanderingThoughts
(About the blog)

Full index of entries
Recent comments

This is part of CSpace, and is written by ChrisSiebenmann.
Twitter: @thatcks

* * *

Categories: links, linux, programming, python, snark, solaris, spam, sysadmin, tech, unix, web

This is a DWiki.
GettingAround
(Help)

Search:

Page tools: View Source, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Feb 3 01:44:38 2006
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.