The basics of
/etc/mailcap on Ubuntu (and Debian)
One of the things that is an issue for any GUI desktop and for many
general programs is keeping track of what program should be used
to view or otherwise handle a particular sort of file, like JPEGs,
.docx files. On Ubuntu and Debian systems, this is handled
in part through the magic file
/etc/mailcap, which contains a bunch
of mappings from MIME types to what should handle them, with various
trimmings. You can also have a personal version of this file in your
home directory as
In the old days when we didn't know any better, installing and
removing programs probably edited
/etc/mailcap directly. These
days the file is automatically generated from various sources,
including from individual snippet files that are stored in
/usr/lib/mime/packages. Various programs drop files in this
directory during package installation and then
magically run to rebuild
/etc/mailcap. One should not confuse
/usr/share/mime/packages; the latter
has XML files that are used by the separate XDG MIME application
(As the update-mime manpage covers, it also uses the
information found in
.desktop files in /usr/share/applications.)
As far as I know, the update-mime manpage is
the sole good source for information about the format of these
little snippets in /usr/lib/mime/packages and the eventual format
of mailcap entries. The format is arcane, with many options and
quite a lot of complex handling, and there is no central software
package for querying the mailcap data; for historical reasons,
everyone rolls their own, with things like the Python
and other parts of this come from the
(For fun, there are multiple generations of mailcap standards. We
start with RFC 1524
from 1993, and then extend from there. On Ubuntu systems, the
doesn't document all of the directives that
update-mime does, for
A single MIME type may have multiple mailcap entries once all of
the dust settles (plus the possibility of wildcard entries as well
as specific ones, for example a 'text/*' entry as well as a
'text/x-tex' one). For example, on our Ubuntu login servers, there
are no less than 7
/etc/mailcap entries for text/x-tex, and 13
for image/png and image/jpeg. In theory people using /etc/mailcap
are supposed to narrow down these entries based on whether or not
they can be used in your current environment (some only work in an
X session, for example) and their listed priorities. In practice the
mailcap parsing code you're using probably doesn't support the full
range of complexity on a current Ubuntu or Debian system, partly
because features have been added to the format over time, and it
may simple pick either the first or the last mailcap entry that
The Freedesktop aka XDG specifications have their own set of MIME association tracking standards, in the Shared MIME database specification and the MIME applications associations specification. These are used by, among other things, the xdg-utils collection of programs, which is how at least some GUI programs decide to handle files. I believe that these tools don't look at /etc/mailcap at all, but they do use MIME type information from .desktop files in /usr/share/applications and the XML files in /usr/share/mime/packages. They might even interpret it in the same way that update-mime does. The XDG tools and MIME associations all assume that you're using a GUI; they have no support for separate handling of a text mode environment.
Any particular GUI program might rely on the XDG tools, use mailcap, or perhaps both, trying XDG and then falling back on mailcap (parsed with either its own code or some library). A text mode program must use mailcap. I'm not sure how self-contained environments like Firefox and Thunderbird work, much less Chrome.
(See also the Arch Wiki page on default applications.)
An appreciation for Cinnamon's workspace flipping keyboard shortcuts
When I first started using Cinnamon (which was back here for serious use), I thought of it just as the closest good thing I could get to the old Gnome 2 environment (as Gnome 3 is not for me). Over time, I've come to appreciate Cinnamon for itself, and even propagate aspects of Cinnamon back into my desktop fvwm setup (such as keyboard control over window position and size). One of the little Cinnamon aspects I now appreciate is its slick and convenient keyboard handling of what I would call virtual screens and Cinnamon calls workspaces.
As basic table stakes, Cinnamon organizes workspaces sequentially and lets you move left and right through them with Ctrl + Alt + Right (or Left) Arrow. By default it has four workspaces, which is enough for most sensible people (I'm not sensible on my desktop). Where Cinnamon gets slick is that it has an additional set of keyboard shortcuts for moving to another workspace with the current window, Ctrl + Alt + Shift + Left (or Right). It turns out that it's extremely common for me to open a new window on my laptop's relatively constrained screen, then decide that things are now too cramped and busy in this workspace and I want to move. The Cinnamon keyboard shortcuts make that a rapid and fluid operation, and I can keep moving the window along to further workspaces by just hitting Right (or Left) again while still holding down the other keys.
(As I've experienced many times before, having this as an easy and rapid operation encourages me to use it; I shuffle windows around this way on my laptop much more than I do on my desktops, where moving windows between virtual screens is a more involved process that generally requires multiple steps.)
Every so often I've thought about trying to create a version of this keyboard shortcut in fvwm, but so far I haven't seen a good way to do it. Although fvwm has functions and supports some logic operations, the feature's actually got a variety of challenges in fvwm's model of windows and what the current window can be. I'm pretty sure that if I looked at the actual Cinnamon code for this, it would turn out to be much more complicated than you'd expect from such a simple-sounding thing.
(I already have a keyboard shortcut for just moving to a different virtual screen; the tricky bit in fvwm is taking the current window along with me when (and only when) it's appropriate to do so given the state of the current window. I suppose the easy way to implement this is to assume that if I hit the 'take the window with me' shortcut, I've already determined that what fvwm considers the current window should be moved to the target virtual screen and my fvwm function can just ignore all of the possible weird cases.)
The uncertainty of an elevated load average on our Linux IMAP server
We have an IMAP server, using Dovecot on Ubuntu 18.04 and with all of its mail storage on our NFS fileservers. Because of historical decisions (cf), we've periodically had real performance issues with it; these issues have been mitigated partly through various hacks and partly through migrating the IMAP server and our NFS fileservers from 1G Ethernet to 10G (our IMAP server routinely reads very large mailboxes, and the faster that happens the better). However, the whole experience has left me with a twitch about problem indicators for our IMAP server, especially now that we have a Prometheus metrics system that can feed me lots of graphs to worry about.
For a while after we fixed up most everything (and with our old
OmniOS fileservers), the IMAP
server was routinely running at a load average of under 1. Since
then its routine workday load average has drifted upward, so that
a load average of 2 is not unusual and it's routine for it to be
over 1. However, there are no obvious problems the way there used
to be; '
top' doesn't show constantly busy IMAP processes, for
example, indicators such as the percentage of time the system spends
in iowait (which on Linux includes waiting for NFS IO) is consistently low, and our IMAP stats
monitoring doesn't show any clear slow commands the way it used to.
To the extent that I have IMAP performance monitoring, it only shows
slow performance for looking at our test account's INBOX, not really
(All user INBOXes are in our NFS
/var/mail filesystem and some
of them are very large, so it's a really hot spot and is kind of
expected to be slower than other filesystems; there's only really
so much we can do about it. Unfortunately we don't currently
have Prometheus metrics from our NFS fileservers, so I can't easily tell if there's some
obvious performance hotspot on that fileserver.)
All of this leaves me with two closely related mysteries. First, does this elevated load average actually matter? This might be the sign of some real IMAP performance problem that we should be trying to deal with, or it could be essentially harmless. Second, what is causing the load average to be high? Maybe we frequently have blocked processes that are waiting on IO or something else, or that are running in micro-bursts of CPU usage.
(eBPF based tracing might be able to tell us something about all of this, but eBPF tools are not really usable on Ubuntu 18.04 out of the box.)
Probably I should invest in developing some more IMAP performance measurements and also consider doing some measurements of the underlying NFS client disk IO, at least for simple operations like reading a file from a filesystem. We might not wind up with any more useful information than we already have, but at least I'd feel like I was doing something.
The case of mysterious load average spikes on our Linux login server
We have a Linux login server that is our primary server basically by default; it's the first one in numbering and the server a convenient alias is pointed to, so most people wind up using it. Naturally we monitor its OS level metrics as part of our Prometheus setup, and as part of that a graph of its load average (along with all our other interesting servers) appears on our overview Grafana dashboard. For basically as long as we've been doing this, we've noticed that this server experiences periodic and fairly drastic short term load average spikes for no clear reason.
A typical spike will take the 1-minute load average from 0.26 or
so (the typical load average for it) up to 6.5 or 7 in a matter of
seconds, and then immediately start dropping back down. There seems
to often be some correlation with other metrics, such as user and
system CPU time usage, but not necessarily a high one. We capture
top output periodically for reasons beyond the scope of
this entry, and these captures have never shown anything in particular
even when they capture the high load average itself. The spikes
happen at all times, day or night and weekday or weekend, and don't
seem to come in any regular pattern (such as every five minutes).
The obvious theory for what is going on is that there are a bunch
of processes that have some sort of periodic wakeup where they do
a very brief amount of work, and they've wound up more or less in
sync with each other. When the periodic wakeup triggers, a whole
bunch of processes become ready to run and so spike the load average
up, but once they do run they don't do very much so the log-jam
clears almost immediately (and the load average immediately drops).
Since it seems to be correlated with the number of logins, this may
be something in systemd's per-login process infrastructure. Since
all of these logins happen over SSH, it could also partly be because
we've set a
ClientAliveInterval in our sshd_config so sshd
likely wakes up periodically for some connections; however, I'm not
clear how that would wind up in sync for a significant number of
I don't know how we'd go about tracking down the source of this without a lot of work, and I'm not sure there's any point in doing that work. The load spikes don't seem to be doing any harm, and I suspect there's nothing we could really do about the causes even if we identified them. I rather expect that having a lot of logins on a single Linux machine is now not a case that people care about very much.
I'm likely giving up on trying to read Fedora package update information
Perhaps unlike most people, I apply updates to my Fedora machines
through the command line, first with
yum and now with
part of that, I have for a long time made a habit of trying to read
the information that Fedora theoretically publishes about every
package update with '
dnf updateinfo info', just in case there was
a surprise lurking in there for some particular package (this has
sometimes exposed issues, such as when I discovered that Fedora
maintains separate package databases for each user).
Sadly, I'm sort of in the process of giving up on doing that.
The overall cause is that it's clear that Fedora does not really
care about this update information being accurate, usable, and
accessible. This relative indifference has led to a number of
specific issues with both the average contents of update information
and to the process of reading it that make the whole experience
both annoying and not very useful. In practice, running '
updateinfo info' may not tell me about some of the actual updates
that are pending, always dumps out information about updates that
aren't pending for me (sometimes covering ones that have already
been applied, for example for some kernel updates), and part of
the time the update information itself isn't very useful and has
'fill this in' notes and so on. The result is verbose but lacking
in useful information and frustrating to pick through.
The result is that '
dnf updateinfo info' has been getting less
and less readable and less useful for some time. These days I skim
it at best, instead of trying to read it thoroughly, and anyway
there isn't much that I can do if I see something that makes me
wonder. I can get most of the value from just looking at the package
list in '
dnf check-update', and if I really care about update
information for a specific package I see there I'm probably better
off doing '
dnf updateinfo info <package>'. But still, it's a hard
to let go of this; part of me feels that reading update information
is part of being a responsible sysadmin (for my own personal
Some of these issues are long standing ones. It's pretty clear that
the updateinfo (sub)command is not a high priority in DNF as far
as bug fixes and improvements go, for example. I also suspect that
some of the extra packages I see listed in '
dnf updateinfo info'
are due to DNF modularity
I'm seeing updateinfo for (potential) updates from modules that
either I don't have enabled or that '
dnf update' and friends are
silently choosing to not use for whatever reasons. Alternately they
are base updates that are overridden by DNF modules I have enabled;
it's not clear.
(Now that I look at '
dnf module list --enabled', it seems that I
have several modules enabled that are relevant to packages that
updateinfo always natters about. One update that updateinfo talks
about is for a different stream (libgit2 0.28, while I have the
libgit2 0.27 module enabled), but others appear to be for versions
that I should be updating to if things were working properly.
Unfortunately I don't know how to coax DNF to show me what module
streams installed packages come from, or what it's ignoring in the
main Fedora updates repo because it's preferring a module version