Wandering Thoughts archives

An interesting internal Django error we just got

As a result of someone trying to either exploit or damage it, our account request system just notified us that it had hit an internal exception in the depths of Django. While that's not too great, what was really interesting was the specific exception and where it happened; it boiled down to:

[...]
File ".../django/db/backends/sqlite3/base.py", line XXX, in execute
  return Database.Cursor.execute(self, query, params)
OverflowError: long too big to convert

Wait, what?

It turns out that the cause of this is (to me) very interesting and also completely explicable once we trace down the layers. We need to start at the actual form. Among other things, this presents a <select> element with the possible values drawn from the database. How Django implements this in our case is that the text of each option is under our control but the HTML form 'value' for each option is an integer (which happens to be the database row's internal primary key). Ie, it looks like this:

[...]
<option value="5">Blah blah</option>
<option value="6">More blah</option>
[...]

Our attacker edited the HTML (I believe using Firefox's developer tools) to provide a really absurdly large value for the option that they picked; for example, one attempt had '518446744073709551616' for this form element. Because Django is a modern web framework it of course does not simply trust this submitted value; instead it validates it and in the process turns it into a proper reference to an ORM object representing the particular database row. If I'm reading the code right, this validation is done ultimately by making a SQL query to look up the row given its primary key (in the process this validates that it's among the set you provided).

This is where we descend down the layers to the SQLite driver. Because the SQLite driver is a good modern database driver, it uses SQL placeholders. Since the primary key field is an integer, this means that the SQLite driver must convert the value passed down by Django to an actual integer in order to pass it as a placeholder, and not just a Python integer but an actual C-level long. As it happens, Django has not passed down the raw string value but has already called int() on the hacked up string, which has given us a Python long integer. This long integer is of course far too big to fit into the C-level long that the SQLite driver requires and the driver notices, giving us this OverflowError (well, it turns out that it is the core Python code that notices, but close enough).

(If you modify the form to something that is not an integer at all, Django detects it at a much higher level and rejects the form cleanly.)

I find this an interesting error partly because of how the low level issues involved show through. A whole cascade of things had to combine together to create this error, including Python's unification of ints and longs, and it is the sort of really obscure corner case that can easily slip through and be overlooked.

(Since it can be triggered from the outside it's probably worth reporting it as a Django bug, but I need to verify that it's still there in the current version. We're a bit behind by now for various reasons.)

PS: We found out about this problem because one of Django's cool features is that it can be set to email you reports about uncaught exceptions such as this. The reports include not just the backtrace but also things like the form POST parameters, which was vital in this case. Without the POST parameters I would have been totally lost; with them, once I started looking the absurd values of this particular form field jumped right out at me.