Accidentally shooting yourself in the foot in Python
Recently, I stumbled over a small issue in Python's cgi module that is a good illustration of how unintended consequences in Python can wind up shooting you in the foot.
The cgi module's main purpose is to create a dictionary-like
object that contains all of the parameters passed to your CGI program
POST HTTP command. DWiki uses it roughly like this:
form = cgi.FieldStorage() for k in form.keys(): ... stuff ...
Then one day a cracker tried an XML-RPC based exploit against DWiki and
this code blew up, getting a
TypeError from the
This is at least reasonable, because an XML-RPC
POST is completely
different than a form
POST and doesn't actually have any form
parameters. (TypeError is a bit strong, but it did ensure that DWiki
No problem; I could just guard the
form.keys() call with an '
form: return'. Except that the '
not form' got the same TypeError.
Which is startling, because you don't normally expect '
not obj' to
throw an error.
This surprising behavior of the cgi module happens through three steps. First, Python decides whether objects are True or False like this:
- if there is a
__nonzero__method, call that.
- if there is a
__len__method, a zero length is False and otherwise you're True (because Python usefully makes collections false if they're empty and true if they contain something).
- if there is neither, you're always True.
As a dictionary-like thing, FieldStorage defines a
in the obvious way:
def __len__(self): return len(self.keys())
Finally, FieldStorage decided to let instances represent several
different things and that calling
.keys() on an instance that wasn't
dealing with form parameters should throw TypeError. (This is more
sensible than this description may make it sound.)
Apart from a practical illustration of unintended consequences and complex interactions, what I've taken away from this is to remember than __len__ on objects is used for more than just the len() function. (Other special methods also have multiple uses.)
Sidebar: so how did I solve this?
My solution was lame:
try: form.keys() except TypeError: return
I suspect that the correct solution is to check
form.type to make
sure that the
POST came in with a Content-Type header of
'application/x-www-form-urlencoded'. (Except I don't know enough to
know if all
POSTs to DWiki will always arrive like that. Ah, HTTP,
we love you so.)