Python 3 module APIs and the question of Unicode conversion errors
I have a little Python thing to log MIME attachment type information from Exim; as has been my practice for some time, it's currently written for Python 2. For various reasons beyond the scope of this entry, today I decided to see if I could get it running with Python 3. In the process, I ran into what I have decided to consider a Python 3 API design question.
My Python program peers inside tar, zip, and rar archives in order to get the extensions of files inside them, using the tarfile, zipfile, and rarfile modules for this; the first two are in the standard library, the third is a PyPi addon. This means that the modules (and I) are dealing with file names that may come in from the outside world in essentially any encoding, or even none (as some joker may have stuffed random bytes into the alleged filenames, especially for tar archives). So, how do the modules behave here?
Neither tarfile nor zipfile make any special comments about the
file names that they return; in Python 3, this means that they
should at least be returning regular (Unicode) strings all of the
time, with no surprise bytestrings if they can't decode things.
Rarfile supports both Python 2.7 and Python 3, so it sensibly
explicitly specifies that its filenames are always Unicode. Tarfile
has an explicit section on Unicode issues that
answers all of my questions; the default behavior is sensible and
you can change it if you want. Both zipfile and rarfile are
more or less silent about Unicode issues for reading filenames in
archives. Code inspection of
zipfile.py in Python 3.5 reveals
that it makes no attempt to handle Unicode decoding errors when
decoding filenames; if any occur, they will be passed up to you
(and there is nothing you can do to set an error handling strategy).
Rarfile attempts several encodings and if that fails, tries the
default charset with a hard-coded 'replace' error handler.
(On the other hand, many ZIP archives should theoretically not have filename decoding errors because the filenames should explicitly be in UTF-8 and zipfile decodes them as such. But I'm a sysadmin and I deal with network input.)
These three modules represent three different approaches to handling potential Unicode decoding errors in Python 3 in your API (and to documenting them); just assume that you're working in a properly encoded world (zipfile), fully delegate to the user (tarfile), or make a best effort and then punt (rarfile). Since two of these are in the standard library, I'm going to assume that there's no consensus so far on the right sort of API here among the Python 3 community.
My personal preference is for the tarfile approach, since it clearly is the most flexible and powerful. However I think there's a reasonably coherent argument for the zipfile approach under some situations, namely that the module is (probably) not designed to deal with malformed ZIP archives in general. I'd certainly like it if the zipfile module didn't blow up on malformed ZIP archives, but my usage case is a somewhat odd one; most people aren't parsing potentially malicious ZIP archives.
(Tarfile has no choice here, as there is no standard for what the filename encoding is in tar archives. A correctly formed ZIP archive that says 'this filename is UTF-8' should always have a filename that actually is UTF-8 and will decode without errors.)
The various IDs of disks, filesystems, software RAID, LVM, et al in Linux
Once upon a time, you put simple
/dev/sdX names in your
These days that's boring and deprecated, and so there are a large
number of different identifiers that you can use here. Since I just
confused myself on this today, I want to write down what I know and
looked up about the various levels and sorts of identifiers, and
where they come from. What I care about here are identifiers that
are tied to a specific piece of hardware or data, instead of where
that hardware is plugged into the system
(or the order in which it's recognized during boot, which can totally
jump around even when no hardware changes).
Some filesystems have labels, or at least can have labels, and years
ago it was common for Linux installs to set labels on your filesystems
and use them in
LABEL=.... This has fallen out
of favour since then, for reasons I can only theorize about. ExtN
is one such filesystem, and labels can be inspected (and perhaps
e2label. Modern Linux distributions seem to no longer
set a label on the extN filesystems that they create during
installation. Just to confuse you, extN filesystems also keep track
of where they were last mounted (or are mounted), which is different
from the extN label, and some tools will present this as the 'name'
of the filesystem.
e2label is effectively obsolete today; you should use
Many filesystems have UUIDs, as
do swap areas, software RAID arrays, LVM objects, and a number of
other things. UUIDs are what is commonly used in
days, and can be displayed with eg '
lsblk -fs'. The
is generally the master source of information about any particular
thing. Like labels, UUIDs are embedded in the on-disk metadata of
various things; for extN filesystems the filesystem UUID is in the
superblock, for example. Where software RAID stores its metadata
varies and can matter for some
things. Note that software RAID has both a UUID for the overall
array and a device UUID for each physical device in the array.
blkid will report, GPT partitions themselves
have a partition label and a theoretically unique partition UUID. These can also be used in
/etc/fstab, per the
fstab manpage, but you probably don't want
to. The GPT UUID is stored as part of the GPT partition table, not
embedded in the partition itself.)
Physical disks have serial numbers (and World Wide Names) that theoretically
uniquely identify them. Where they're accessible, Linux reads these
via SCSI, SAS, iSCSI, SATA, and so on inquiry commands, and uses
this information to populate
/dev/disk/by-id. In addition to
actual disks, generally anything that appears as a disk-like device
with a UUID (or a name) will also show up in
Thus you can find things like software RAID arrays (by name and
UUID), LVM physical volumes, and LVM logical volumes (by name and
(I believe that some USB disk enclosures don't pass through the necessary stuff for Linux to get the disk's serial number.)
Sometimes this can get confusing because the same object winds up
with multiple IDs at different levels. A software RAID array or a
LVM logical volume that contains an extN filesystem has both a UUID
for the filesystem and a UUID for the array or volume, and it may
not be clear which UUID you're actually using unless you look in
blkid is generally fairly clear, fortunately;
lsblk's default output is not so much from what I've seen.
(If you're looking at an
/etc/fstab generated by an installer
or the like, they generally use the filesystem UUID.)