2008-11-13
What the members of a Unicode conversion error object are
The codecs module's
register_error
function has a rather scanty description of what
error handlers are called with. Since I have just dug into this,
here is what the members are for at least UnicodeDecodeError:
object |
The string that is being decoded to Unicode. |
encoding |
The encoding that the string is (theoretically) in. |
start |
The index in object of the first character
that could not be decoded. |
end |
One past the end character of this decoding problem. |
reason |
The human-readable error text. |
The substring object[start:end]
is the character or character sequence
that had decoding problems (perhaps I should say 'byte', but this is not
yet Python 3K). You get a character sequence instead of a character in
situations where the first character is a valid start of a multi-byte
sequence, but subsequent characters have an error.
For example, in a theoretical UTF-8 encoded string, the first two
bytes of the three-byte sequence 0xe0 0x81 0x58 are valid parts of
a three-byte encoding, but the third byte is not. You would get a
UnicodeDecodeError object where start
pointed to 0xe0, the first byte,
and end
was just past 0x58, the third.
Also worth noting is that when things are being decoded to Unicode, the first element of the tuple your error handler returns (assuming that you want to keep going) has to be a unicode string.
Given all of this, we can put together a really simple decoding error handler that just replaces undecodable bytes with backslashed hex versions of themselves:
def bsreplacer(uerr): c = uerr.object[uerr.start] return (u"\\x%x" % ord(c), uerr.start+1) import codecs codecs.register_error("bsreplace", bsreplacer)
Note that we are playing fast and loose with multi-byte sequences; instead of handling the entire sequence, we just replace the first byte and restart decoding after it. In some situations this can malfunction and produce garbled output.
(One would think that the standard "backslashreplace" error handler would already do this, but unfortunately it doesn't handle decoding errors, only encoding errors.)