Wandering Thoughts archives

How I should have done password crypto for DWiki

DWiki (the software behind WanderingThoughts) uses cryptography for the traditional web application things, namely login passwords and login cookies; it stores hashed versions of login passwords, and it signs its login cookies. I had very little idea what I was doing when I wrote that code, so DWiki's current use of cryptography is better off not discussed in polite company; instead, let's discuss how it should work.

(I was so stupid that I didn't use HMAC, despite it being in the Python library, because I thought it was too complicated and I preferred something simpler that I could fully understand. Well, the joke's on me.)

First, you need to store encrypted passwords. The basic minimum requirement is to generate a unique per-password salt with os.urandom(), do hmac.new(salt, passwd, hashlib.sha1), and store both the HMAC output and the salt. Per Matasano Chargen, using bcrypt is much better; however, there is no Python module for it in the standard library, so if you're that worried you'd need to build the bcrypt module from pypi. As far as I know, there is no standard library support for better (ie significantly slower) hashes than SHA1.

(You might ask why not SHA512? According to openssl speed, it's only two to three times slower than SHA1, ie still pretty fast, and it means you have very large password hashes. If you're worried that much about security, go with bcrypt. Also, I lack enough crypto knowledge to know if repeatedly SHA1-ing the HMAC digest result will damage your security. If it doesn't, adding some extra SHA1 rounds will at least vaguely mimic some of bcrypt's advantages and completely erase SHA512's.)

To sign and authenticate cookies, use HMAC (with SHA1) on anything you need to authenticate plus a global secret key (any random value will do; I got DWiki's from /dev/urandom), and put in the login cookie both the plaintext and the authenticator digest. If you don't care about how long people have valid cookies, you can just sign the login name; if you do care, you need to sign both the login name and the expiry time. Make sure to sign exactly what you put in the login cookie (minus the signature, obviously).

(By now I am sufficiently paranoid that I would include the delimiter between the plaintext and the signature in what I sign, just in case.)

(Disclaimer: this may turn out to be horribly naive in N years, just like the last time around.)

A Python safety tip: Do notice that things can throw exceptions

Recently, I discovered and fixed a long-standing bug in DWiki. The bug itself was simple, but typical of a certain kind of error that I keep making in Python: I had forgotten that the Cookie module could throw an exception, so I didn't catch it.

(Well, my general error code logged it, but I mean DWiki itself didn't catch it and do anything sensible; in this case, either ignoring the broken HTTP cookie header or perhaps reporting a security error.)

This bug lingered for so long because the Cookie module more or less only throws errors for invalidly formatted cookies, and invalidly formed cookies are often very rare. Most browsers make sure to only send valid cookies themselves and are very careful about not sending you other people's cookies, so under many circumstances it can take either a badly broken client or someone deliberately trying to attack you before you see a badly formed cookie. The net result is that you could run a web application for years before this particular code path gets triggered.

(This is what happened to DWiki, and I'm not sure if it was an attack or just a really, really bad client. Possibly both.)

I have no real excuse for this; the Cookie module's documentation certainly covered the CookieError exception at the time that I wrote the relevant DWiki code. I just didn't read that bit of the documentation. (The current module documentation contains a more prominent caution about this, perhaps caused by other people also skipping over the previous version.)

This sort of mistake is almost inevitable in any language that doesn't force you to explicitly code error handlers (either by forcing you to explicitly handle exceptions, or explicitly including a status return value and making you do something with it); Python at least insures that the error is handled, one way or another. Doing better is probably the domain of code analysis tools like pylint, although detecting this sort of thing might take fairly serious analysis work unless people start annotating code and modules with information about what exceptions they raise.

(I don't believe that test driven development would have caught this error, since I doubt I would have thought to include a test with a corrupted cookie. If I had been thinking about that possibility in the first place, I would have found the exception in the documentation.)

Three ways to get tracebacks in your CGI Python application

There are at least four ways to get Python tracebacks in your CGI or web-app in case something goes wrong; the easy but wrong way, the more or less right but inconvenient way, the flawed way, and the good way.

The easy but wrong way is 'import cgitb; cgitb.enable()'. The reason this is the wrong way (except when you yourself are developing your program) is that this dumps error tracebacks on the users of your application. Ignoring any information disclosure issues, reporting errors to the users is both wrong and pointless; the odds that users will actually bother to save the traceback and report it to you are somewhere between low and nil.

The more or less right way is to use the cgitb module to log tracebacks to files in some directory somewhere, and not display the traceback to your users. This has two drawbacks; first, your users get at best a very minimal error message, and second you have to monitor the directory to check for problems.

(It may be helpful to know that the cgitb module still produces a very brief report to standard output even if it is set to not display the traceback itself. If you use HTML format tracebacks, this report is in HTML, cleverly formatted to be valid either as HTML or the entire output of a CGI program. If you use plain text tracebacks, this report is in plain text and probably won't display in people's browsers, which may be considered a feature.)

The flawed way is to not do anything special, so that you CGI just writes a regular Python traceback to standard error and thus to your web server's error log. The drawback of this approach is that it makes it hard to pick things out of the error log. While web servers are getting better about making sure every line of the error log has basic information (such as timestamp and request source), your messages won't have things like the program that produced them. Among other issues, this can make it hard to find out that your program is having problems.

The good way is to catch the traceback and reformat it to have a distinctive structure. What I do is prefix each line of the traceback with '<program>: T: ', and surround the whole traceback with a start and an end line to make it easier to pick out the whole thing. You can set this up as the system exception hook, making this approach just as easy to use as the cgitb module.

(Having just reminded myself how simple the good way is to use, I think I should go plug it into some of our internal CGIs (which are currently using the more or less right way).)

Sidebar: my stderrtb module code

Here is the entire code I use for this in DWiki:

import sys
from traceback import format_exception

def stderr(msg):
    sys.stderr.write("%s: %s\n" % (sys.argv[0], msg))

def print_except(t, inf, tr):
    stderr("Python internal error:")
    for e in format_exception(t, inf, tr):
        lines = e.split("\n")
        for l in lines:
            if not l:
                continue
            stderr("T: %s" % l)
    stderr("-- traceback finished")

def enable():
    sys.excepthook = print_except

Call the enable() routine to turn this on. Note that this needs no support from your application; it works just like the cgitb module does. You can reuse print_except if you need more sophisticated error handling and recovery in your application.