Chris's Wiki :: blog/python/SemiEvilIntrospection

Using Python introspection for semi-evil

November 9, 2005

One of the things I write in Python is network daemons. Because it works so nicely, network daemons usually take input as text lines that start with a command word and have whitespace separated arguments. There's a certain amount of eye-rolling tedium in writing code to check that the command word is valid and has the right number of arguments. When I wrote a program that does a lot of this, the tedium finally got to me and I sort of snapped and automated all of the validation through Python's introspection features.

The program's basic structure has an object for each connection. Each command is handled by a method function on the object, and the functions all take normal argument lists. (So they are not passed the command line as a big list or anything.)

The name of each command method is 'op_<command>', eg 'op_sample' to handle the 'sample' command. To check whether the line's first word was a valid command, I just looked to see if the connection's object had an attribute with the appropriate name.

(This is a fairly common pattern in Python; see, for example, how BaseHTTPServer handles dispatching GET and POST and so on.)

To check that the number of arguments was right, I reached into the handler function I'd just found and fished out how many arguments it expected to be called with (compensating for the additional 'self' argument that method functions get). This isn't at all general, but I didn't need generality; the network daemon's commands all have a fixed number of arguments.

The code wound up looking like this:

# in a class:
def op_sample(self, one, two):
  ...

def dispatch(self, line):
  # Do we have a line at all?
  n = line.split()
  if not n:
    return False
  cword = 'op_' + n[0]

  # Valid command word?
  cfunc = getattr(self, cword, None)
  if not cfunc:
    return False

  # Right argument count?
  acnt = cfunc.func_code.co_argcount
  if acnt != len(n)
    return False

  return cfunc(*n[1:])

(The real version used more elaborate error handling for 'empty line', 'no such command', and 'wrong number of arguments'.)

Normally I would have to account for the extra self argument in the function's argument count. But in this code the n list has one extra element (the command word) too, so it balances out. This has the subtle consequence that you can't make op_sample a staticmethod function, because then it would have the wrong argument count.

(I did say this was semi-evil.)

Comments on this page:

From 192.88.60.254 at 2005-11-09 14:41:31:

Doesn't introspection like this kill your performance?

I'll note that I considered using something similar when I rewrote wikirend.py, but since one of the two main goals of that rewrite was performance, instead I went with this pattern:
filter_routines = {}
def blank_handler(self, tok):
        pass
filter_routines['blank'] = blank_handler

def starsep_handler(self, tok):
        self.result.append('<p align="center">* * *</p>\n')
filter_routines['starsep'] = starsep_handler

def hr_handler(self, tok):
        self.result.append('<hr/>\n')
filter_routines['hr'] = hr_handler

def header_handler(self, tok):
        hlevel = min(len(tok[3].group(1)),6)
        htext = tok[3].group(2)
        hdtag = 'h%d' % hlevel
        self.handle_begin('begin', hdtag)
        self.handle_text('text', htext)
        self.handle_end('end', hdtag)
filter_routines['header'] = header_handler
That is, even though I named all the _handler methods similarly, and could have used introspection like this on each token to determine the method to call, I chose to fill up an extra dictionary so that I would not be constantly going through reflection to get the appropriate method.

Hrm. In Python this introspection may in fact be nothing more than a dictionary lookup anyway, so I may not have saved much. However, I don't think that the extra filter_routines line is much tedium to contend with. I can see how it might get a bit more tedious if I needed to put more information into the dictionary about argument types, etc.

Note that in Python 2.4, much of this could be handled by annotations to the individual handler procedures.

By cks at 2005-11-09 17:08:40:

This sort of introspection only adds a few string concatenations and dictionary/method lookups. In a typical network daemon (and certainly for this one), this is dwarfed by things like the slow speed of network IO and disk IO.

Written on 09 November 2005.

«	Fun with DNS: a semi-obscure problem

How doing relative name DNS lookups can shoot you in the foot