Industrial strength Python

September 7, 2006

It's not uncommon to have people harsh on Python and similar languages as being 'scripting languages', not suitable for serious jobs because they're not fast enough or they use too much memory, or the like. I have to beg to differ, because I have a great counter-example.

We use a SMTP frontend daemon to check connections before we bother to start a real SMTP conversation. This program does our greylisting, our DNS blocklist checking, connection count limiting, and a number of other checks. And it's written in Python, for various reasons.

Last week it handled 1.4 million connections, with over 700,000 coming in one day. For those 24 hours, it was handling just over 8 connections a second. Yes, it used a bit of CPU time to do this (it seems to have averaged a bit under 0.06 CPU seconds per connection), but modern machines generally have a lot of CPU to spare.

Nor did it gobble memory to do this; at the end of the week, the process was using 20 megabytes of virtual memory, with an 11 megabyte resident set size. This is up from its starting size, which is around 8 megabytes with 5.5 megabytes of RSS, but the frontend is remembering the first and last connection times of most every IP address that ever talked to it; last week it was tracking almost 53,000 of them. A version of the frontend written in C (or maybe Java) would probably use less memory. But the Python version's memory usage is not over the top or excessive for a modern machine, and it's not leaking.

I won't claim that writing industrial strength Python like this is completely easy; you do have to pay attention to detail and watch out for various things, and I certainly got my hands dirty poking around down in the depths of Python's object management in the process of making sure that I was using as little memory as possible. But it's not hugely difficult, and a lot of it is common sense.

(And to a fair extent you're going to have to do this no matter what language you use; industrial strength programs require attention to details, period. Different languages just require you to pay attention to different bits.)

Written on 07 September 2006.
« Some wise words from Henry Spencer on backups
Link: IRON File Systems »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Thu Sep 7 22:29:05 2006
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.