Modules should never raise core Python exceptions

September 6, 2016

Today I discovered that the zipfile module raises RuntimeError if you ask it to read a file from an encrypted ZIP archive and do not give it the right password (or don't give it a password at all). I wasn't happy. Apparently it isn't obvious to the people who maintain Python's standard library (and who updated it for Python 3) why this is a terrible idea (not that I'm surprised about that).

Here is a simple rule for module writers:

Your module should never raise an exception that the interpreter can raise for plain Python code.

The reason is simple, as illustrated here. RuntimeError can be raised by the Python interpreter under various circumstances, if various things go sufficiently badly wrong. Since the zipfile module raises it too, if you are trying to do something with a ZIP archive that may or may not be encrypted, you must use an over-broad try:

  filedata =
except RuntimeError as e:

When that RuntimeError exception fires, how do you know whether the ZIP archive is encrypted, whether the zipfile module has a bug that triggers a RuntimeError from Python, or whether the the Python interpreter ran into an internal runtime error? The answer is that you basically don't. You have to cross your fingers and assume. It's probably an encrypted zipfile, so let's go with that.

(Life would be slightly better if the zipfile module had an interface that told you whether files in a ZIP archive were encrypted or not, but of course it doesn't have a documented way to do that. It doesn't even have an internal constant for the flag bit involved.)

If you have to put additional code inside the try, things get worse for obvious reasons; the more code (either yours or other people's) that's getting called, the more chances that bugs in that code will be the cause of RuntimeError or the like, instead of the exception from the zipfile module that you expect to be the only thing raising that error. We've seen variants on this before, and they can definitely trip you up. Probably not very often, but even one own goal here is more than enough.

This is why modules should never deliberately raise core Python exceptions, exceptions that can be raised by just running plain code. Catching them forces me and you to write overly broad trys because these exceptions are intrinsically broad. When you make up your own exceptions, you insure that they cover exactly and only the errors they should cover, not your errors plus some random set of other ones.

(Yeah, I've sort of said this before.)

PS: The exception to this rule is built-in exceptions that are specifically designed to be raised by user code as a standard signal of problems. An example is NotImplementedError, and it may be the only one that I really agree with.

Comments on this page:

Similar issue: gdbm used to call exit(3) but as of the new version, 1.12.90 (released sometime in July of this year apparently) it no longer does this. It was always my example C library that Did A Bad Thing by inflicting exit(3) on unsuspecting users.

@Kevin Xlib has a good example in XSetIOErrorHandler().

It sets a callback to handle the connection to the X server breaking. It also unconditionally calls exit if the callback returns.

I ran into this in the early days of VMware GSX, because server VMs would die if you had a console attached and the X server crashed/exited. I had optimized the "remote" console to use a local connection if the client and server were on the same machine.

I experimented with longjmp()ing out of the callback, but in the end I just made GSX always use the remote console protocol (a extension of VNC in those days) even when everything was local.

@Nolan: What? That's insane. That's even worse - they let you set an error-handling function and then they call exit(3) anyway. Awesome.

The new gdbm apparently lets you set an error-handling function but does no call exit if it returns.

I went to see if XSetIOErrorHandler still does that and it does. But what I didn't know is that there's an alternative C library for X11 - XCB. Interesting.

It's true that zipfile shouldn't raise RuntimeError in your case (I would make a WrongPasswordError class), but I don't think that your general conclusions, that third-party modules shouldn't raise built-in exceptions is true. Just as one example out of many, say that you're making a dict-like data structure, and someone is trying to access a key that doesn't exist. You should raise KeyError, no?

I can give many more examples.

Written on 06 September 2016.
« Using Magit to selectively discard changes in your git working tree
Why my smartphone is going to be an iPhone »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Tue Sep 6 23:22:13 2016
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.