Wandering Thoughts archives

Python's extra-clever function parameters

I'm not sure where I ran across this, but until recently I wasn't aware that you could pre-explode lists and tuples when you declared what parameters a function takes. That sounds stilted, so I'll give an example:

def foo(a, (b, c, d)):
  print a, b, c, d

This is slightly more compact than the equivalent:

def bar(a, tmp):
  b, c, d = tmp
  print a, b, c, d

Contrary to what it might first look like, foo's second argument doesn't have to be an explicit list or tuple with three elements. Instead, it can be anything that will yield exactly three values when iterated. Want to pass xrange(0, 3), or even {1: 10, 2: 20, 3: 30}? No problem.

(Iterating a dictionary returns the keys (in some random order, so this is not necessarily a useful example).)

This freedom comes with a small drawback. Compare:

>>> foo(10)
TypeError: foo() takes exactly 2 arguments (1 given)
>>> foo(10, (1, 2))
ValueError: unpack tuple of wrong size

(I have elided the tracebacks.)

This is different from a language like Haskell, which has a more pattern-matching approach to this; I believe in Haskell you would get an 'argument mismatch' type of error in both cases. (Haskell looks like the kind of mind-bending neat language that's worth reading about, even if you never write a Haskell program.)

Writing this made me curious enough to look at the actual bytecodes generated for foo and bar (with the dis module). It turns out that they're pretty much identical, because foo starts with a prequel bit to explicitly unpack an explicit but internal second argument into b, c, and d. So foo is pretty much just syntactic sugar, supported by the bytecode compiler, for what bar writes out explicitly.

Although I've used this in code recently, I'm not sure it's entirely a good idea. Sometimes I feel that Python is a little bit too clever for my own good, and that I would be better off if I could resist the temptation to use some of these neat features.

(PS: yes, you can nest pre-exploded parameters. Don't go there.)

More clever (ab)use of and and or

In a comment on the last entry I mumbled yet again how I was annoyed that Python doesn't have an equivalent of C's '? :' conditional expression operator. Then some neurons in the back of my mind woke up and asked 'didn't Guido van Rossum say he was going to add this to Python 2.5?'

In the process of trying to figure out if this was still happening (it is), I read this message (via here) and found out about a trick (used in the standard library no less) that gets most of what I want. For 'A ? B : C', we can write:

A and B or C

(Python precedence rules make this equivalent to '(A and B) or C'. It works because both and and or return the value of the last expression they evaluated.)

This isn't quite a full emulation of ?:, because it gives you C if A is true and B is something that Python will consider false, but it is close enough for many uses. Often my code is such that the A expression isn't true unless B has a non-empty, non-zero value. And if your C is guaranteed to be true, you can always invert the expression.

(This doesn't help the code in the last entry, though, since we can't guarantee that blst will be considered true is the caller supplied it, and our C default value will be considered false.)

Another day, another useful trick, and it's all because of a comment. (Which makes me very glad I bothered to add comments to DWiki.)

Sidebar: Python 2.5 conditional expressions

From Guido van Rossum's message , the syntax will be 'B if A else C' (to use my ordering). I'm not really sure I like this; in general, I don't like out of order syntaxes because they make me backtrack on expressions when I read them. (I touched on this before back here.)

According to the Python 2.5 release schedule, Python 2.5 is supposed to come out around 30 September 2006, so we have a while to wait for this.

A trick for handling mutable default arguments

One of the neat things about programming in Python is finding out about clever little idioms that are obvious in retrospect. My latest find is a little trick for handling mutable function arguments.

As I mentioned back when discussing the consequences of def being an executable statement, the way I'd been handling mutable default arguments was something like:

def foo(a, blst = None):
  if blst is None:
    blst = []

At two lines, this is just verbose enough to grate lightly but irritatingly on my nerves. Compressing the if and the statement onto one line didn't really help, because Python style is fairly firmly against that. Then I read an Ian Bicking blog entry, and the obvious in hindsight shorter version in it jumped out at me:

blst = blst or []

This works because the result of Python's boolean operators is not True or False, but the last expression evaluated. So 'blst or []' is blst if it is considered true and our new anonymous [] otherwise.

But there's a subtle (and fortunately rare) gotcha with this, pointed out by the if it is considered true bit. Namely, if we pass in an explicit argument that's considered to be false, we wind up with blst being [] instead of it. If you do this a lot and care about it, I suppose the way around it is:

_none = object()
def nonnull(*args):
  return [a for a in args \
          if a is not _none][0]

def foo(a, blst = _none):
  blst = nonnull(blist, [])

This is even safe against silent success if someone accidentally passes in an explicit blst argument that turns out to be None (perhaps because they didn't notice another routine returning None).

(There is surely a better name for the helper function than nonnull; I am just bad at naming things.)

Appropriately, the Ian Bicking blog entry I found this in is called Reducing boilerplate code in __init__, although he is talking about something much higher level than this.

Sidebar: another obvious or trick

Today I was writing a case-independent string compare function that needed to have a consistent and stable order if the strings differed only in case. At first I wrote the obvious multi-line function; then I realized I could just write it as:

def ci_cmp(a, b):
  return cmp(a.lower(), b.lower()) or \
         cmp(a, b)

(In my case it wasn't worth memoizing the .lower() results, or equivalently doing a Schwartzian transformation; I'm only sorting a list with a few hundred (short) strings.)

More on regular expression performance

My problem with benchmarking is that it's tedious (and finicky). Good benchmarking mostly consists of twiddling my thumbs while my tests run, often yet again after some minor tweak or improvement or additional case. I can kill days running tests and staring at numbers and not really get anywhere (and sometimes I've had to).

All that said, after my entry on some regular expression performance surprises, I got curious enough to do more detailed micro-benchmarking. With Daniel Martin's assistance I was able to check Perl as well as Python, and we turned up some surprises.

What I looked at was various versions of constant or near-constant alternates; the '(FOO BAR|BAZ BLORP)' case from my earlier entry. I'm primarily interested in the 'miss' case, as before, because it's typical for the sort of text scanning I do.

Things I've found:

• Perl's regexp engine is significantly faster than Python's in most cases. This was especially apparent for simple constant text matches, where Perl was five to ten times faster (depending on the test machine).

• a constant alternative with a prefix, ie 'FOO (BAR|BAZ)', runs much faster than just a constant alternative, even when the constant alternative itself has a constant prefix. This is especially so in Perl, because:

• Perl is catastrophically bad at constant text alternatives; they run at least 90 times slower than the prefix alternate case, and significantly slower than in Python. This really surprised me; I had remembered the situation being bad, but not that bad.

• Adding a constant text lookahead assertion to the constant alternative case (ie, '(?=FOO)(FOO BAR|FOO MORK)') doesn't make a huge improvement in Perl (at most halving the catastrophically bad time), and makes Python slower.

• In Python, prefix alternates run as fast as plain text and faster than two separate regexps. In Perl it's a couple of times slower than plain text and a couple of regexps were still faster.

• In Python, adding more constant alternatives seems to be costless, especially if they have a common prefix. (Python turns out to hoist common constant prefixes out of the alternates during regexp compilation.)
• In Perl, each additional alternative exacts a noticeable penalty (although not a large one compared to the overall costs), more or less the same whether or not they have a common prefix.

• Python is catastrophically worse at a pattern like 'F+OO (BAR|MORK)', managing to run up to 40 times slower than Perl.

Python compiles regexps in Python code, so you can look at this and examine several steps of the result. The most interesting thing to look at is probably the output of sre_parse.parse("<re string>"), which shows how the parser has broken things down and optimized them. (Apparently some additional optimization happens in the compiler too.)

I am a bit peeved at the performance differences between the Perl and Python regexp engines. It used to be that if a Python program was too slow, I could immediately conclude a Perl (or Ruby or etc) version would also be too slow, which simplified life. (Indeed, I could usually conclude that any regexp based version would be too slow, because I figured everyone had highly optimized regexp engines.)

Oh well. As I said myself, measure, don't guess.

(And thank you, Daniel Martin, for writing the Perl version of the benchmark tests.)

Sidebar: The experimental setup details

The test environments:

1. Athlon XP 2100+, Fedora Core 2, Python 2.3.3, perl 5.8.3
2. Intel Pentium 4 2.6GHz, Fedora Core 4, Python 2.4.2, perl 5.8.6

I've put up the rebench.py and rebench.pl test programs, if anyone wants to test themselves.

Some regular expression performance surprises

Part of the fun of performance tuning is all of the unexpected things you find out. Since I spent some of yesterday doing work in the guts of DWiki's DWikiText to HTML conversion, what I've been finding is some regular expression performance surprises.

My big surprise was the relative performance of '[A-Z]', '[A-Z][A-Z]*', and '[A-Z]+' for scanning text where they didn't match (the real character class is more complex). I expected them to be about the same speed; instead, the version using + is at least twice as slow.

Hopefully everyone reading this knows that the Perl regexp engine (which is more or less what's used in Python) checks '|' alternatives strictly left to right, one after the other. Among other things, this means that it is really worth your while to put common alternatives first.

Update: Daniel Martin has pointed out that Python doesn't actually use the Perl regexp engine, just the Perl regexp syntax and semantics. The CPython _sre regexp engine was independantly written by Secret Labs AB. My (bad) error.

But what I suspect is less well known is something I stumbled over some years ago, when looking at the performance of our Usenet anti-spam filters: constant alternatives are significantly slower than separate regular expressions (for non-matching searches, but I suspect that this holds for matching searches too).

Given:

re1 = re.compile("FOO BAR")
re2 = re.compile("BAZ BLORP")
re3 = re.compile("(?:FOO BAR|BAZ BLORP)")

It appears that current Pythons run 're1.search(tstr) or re2.search(tstr)' two or more times faster than 're3.search(tstr)'. The speed advantage grows as the size of the test string increases; using '(...)' instead of '(?:...)' performs no worse.

The results flip if the matches are not constant; '(?:[A-Z] FOO BAR|[A-Z] BAZ BLORF)' is faster than checking '[A-Z] FOO BAR' and '[A-Z] BAZ BLORF' separately. This suggests, not too surprisingly, that the regexp engine has highly tuned constant string matching code that is not triggered by plain string alternatives.

(And '[A-Z] (?:FOO BAR|BAZ BLORP)' is, unsurprisingly, faster still. Also, you can't try to cleverly convert the non-constant versions to constant ones by making the prefix a positive lookbehind assertion, eg '(?<=[A-Z] )FOO BAR'; in fact, you lose significant performance.)

This leads to the obvious moral, which is also the first rule of performance tuning: measure, don't guess.

(I'm still kind of bummed about the constant strings alternative performance drop, but I suppose if I was really annoyed I should be contributing patches to fix it.)

Using the tempfile module to get temporary files

Courtesy of Planet Python, I stumbled over Peter Bengtsson's puzzlement with Python's tempfile module, especially the mkstemp() function. It's an understandable puzzlement, because what's going on is fairly tied to Unix semantics (although Python supports mkstemp() on all platforms).

The tempfile module's mkstemp() is a relatively thin shim over the Unix mkstemp(3) library routine, which exists to securely create temporary files. This is more difficult than it looks; there are all sorts of race conditions, which can lead to security problems.

Update: I'm completely wrong here. Python's mkstemp() doesn't call mkstemp(3); instead it (re)implements the general algorithm and interface. This means that mkstemp() doesn't automatically use any special or necessary tricks done in your platform C library mkstemp(3).

Many of these issues are ultimately caused by the difference between file names and files. Put crudely, people exploit the race conditions by changing what file a given file name points to (there are a number of ways to do this). mkstemp(3) takes steps to guarantee that the file it creates is a new temporary file, but it cannot be absolutely sure that the file name will continue to point to it, so it returns the file (as well as the name).

Because this is Unix, the file is returned in the form of a file descriptor, which the os module deals with. The best way to do anything with this is to turn it into a Python file object with os.fdopen(). Sample code is:

import os, tempfile
def filetemp():
  (fd, fname) = tempfile.mkstemp()
  return (os.fdopen(fd, "w+b"), fname)

This is more or less what the tempfile module's TemporaryFile class does to return a file-like object. (They are real file objects on Unix but wrapped objects on other platforms.)

(The OpenBSD mkstemp(3) manpage has some additional information and commentary that may be useful.)