Accidentally shooting yourself in the foot in Python

Recently, I stumbled over a small issue in Python's cgi module that is a good illustration of how unintended consequences in Python can wind up shooting you in the foot.

The cgi module's main purpose is to create a dictionary-like object that contains all of the parameters passed to your CGI program in the GET or POST HTTP command. DWiki uses it roughly like this:

form = cgi.FieldStorage()
for k in form.keys():
  ... stuff ...

Then one day a cracker tried an XML-RPC based exploit against DWiki and this code blew up, getting a TypeError from the form.keys() call. This is at least reasonable, because an XML-RPC POST is completely different than a form POST and doesn't actually have any form parameters. (TypeError is a bit strong, but it did ensure that DWiki paid attention.)

No problem; I could just guard the form.keys() call with an 'if not form: return'. Except that the 'not form' got the same TypeError. Which is startling, because you don't normally expect 'not obj' to throw an error.

This surprising behavior of the cgi module happens through three steps. First, Python decides whether objects are True or False like this:

• if there is a __nonzero__ method, call that.
• if there is a __len__ method, a zero length is False and otherwise you're True (because Python usefully makes collections false if they're empty and true if they contain something).
• if there is neither, you're always True.

As a dictionary-like thing, FieldStorage defines a __len__ method in the obvious way:

def __len__(self):
  return len(self.keys())

Finally, FieldStorage decided to let instances represent several different things and that calling .keys() on an instance that wasn't dealing with form parameters should throw TypeError. (This is more sensible than this description may make it sound.)

Apart from a practical illustration of unintended consequences and complex interactions, what I've taken away from this is to remember than __len__ on objects is used for more than just the len() function. (Other special methods also have multiple uses.)

Sidebar: so how did I solve this?

My solution was lame:

try:
  form.keys()
except TypeError:
  return

I suspect that the correct solution is to check form.type to make sure that the POST came in with a Content-Type header of 'application/x-www-form-urlencoded'. (Except I don't know enough to know if all POSTs to DWiki will always arrive like that. Ah, HTTP, we love you so.)