Wandering Thoughts archives

2020-02-25

The basics of /etc/mailcap on Ubuntu (and Debian)

One of the things that is an issue for any GUI desktop and for many general programs is keeping track of what program should be used to view or otherwise handle a particular sort of file, like JPEGs, PDFs, or .docx files. On Ubuntu and Debian systems, this is handled in part through the magic file /etc/mailcap, which contains a bunch of mappings from MIME types to what should handle them, with various trimmings. You can also have a personal version of this file in your home directory as ~/.mailcap.

In the old days when we didn't know any better, installing and removing programs probably edited /etc/mailcap directly. These days the file is automatically generated from various sources, including from individual snippet files that are stored in /usr/lib/mime/packages. Various programs drop files in this directory during package installation and then update-mime is magically run to rebuild /etc/mailcap. One should not confuse /usr/lib/mime/packages with /usr/share/mime/packages; the latter has XML files that are used by the separate XDG MIME application specification.

(As the update-mime manpage covers, it also uses the MimeType= information found in .desktop files in /usr/share/applications.)

As far as I know, the update-mime manpage is the sole good source for information about the format of these little snippets in /usr/lib/mime/packages and the eventual format of mailcap entries. The format is arcane, with many options and quite a lot of complex handling, and there is no central software package for querying the mailcap data; for historical reasons, everyone rolls their own, with things like the Python mailcap module. Update-mime and other parts of this come from the mime-support package.

(For fun, there are multiple generations of mailcap standards. We start with RFC 1524 from 1993, and then extend from there. On Ubuntu systems, the mailcap manpage doesn't document all of the directives that update-mime does, for example.)

A single MIME type may have multiple mailcap entries once all of the dust settles (plus the possibility of wildcard entries as well as specific ones, for example a 'text/*' entry as well as a 'text/x-tex' one). For example, on our Ubuntu login servers, there are no less than 7 /etc/mailcap entries for text/x-tex, and 13 for image/png and image/jpeg. In theory people using /etc/mailcap are supposed to narrow down these entries based on whether or not they can be used in your current environment (some only work in an X session, for example) and their listed priorities. In practice the mailcap parsing code you're using probably doesn't support the full range of complexity on a current Ubuntu or Debian system, partly because features have been added to the format over time, and it may simple pick either the first or the last mailcap entry that matches.

The Freedesktop aka XDG specifications have their own set of MIME association tracking standards, in the Shared MIME database specification and the MIME applications associations specification. These are used by, among other things, the xdg-utils collection of programs, which is how at least some GUI programs decide to handle files. I believe that these tools don't look at /etc/mailcap at all, but they do use MIME type information from .desktop files in /usr/share/applications and the XML files in /usr/share/mime/packages. They might even interpret it in the same way that update-mime does. The XDG tools and MIME associations all assume that you're using a GUI; they have no support for separate handling of a text mode environment.

Any particular GUI program might rely on the XDG tools, use mailcap, or perhaps both, trying XDG and then falling back on mailcap (parsed with either its own code or some library). A text mode program must use mailcap. I'm not sure how self-contained environments like Firefox and Thunderbird work, much less Chrome.

(See also the Arch Wiki page on default applications.)

UbuntuMailcapBasics written at 00:20:38; Add Comment

2020-02-21

An appreciation for Cinnamon's workspace flipping keyboard shortcuts

When I first started using Cinnamon (which was back here for serious use), I thought of it just as the closest good thing I could get to the old Gnome 2 environment (as Gnome 3 is not for me). Over time, I've come to appreciate Cinnamon for itself, and even propagate aspects of Cinnamon back into my desktop fvwm setup (such as keyboard control over window position and size). One of the little Cinnamon aspects I now appreciate is its slick and convenient keyboard handling of what I would call virtual screens and Cinnamon calls workspaces.

As basic table stakes, Cinnamon organizes workspaces sequentially and lets you move left and right through them with Ctrl + Alt + Right (or Left) Arrow. By default it has four workspaces, which is enough for most sensible people (I'm not sensible on my desktop). Where Cinnamon gets slick is that it has an additional set of keyboard shortcuts for moving to another workspace with the current window, Ctrl + Alt + Shift + Left (or Right). It turns out that it's extremely common for me to open a new window on my laptop's relatively constrained screen, then decide that things are now too cramped and busy in this workspace and I want to move. The Cinnamon keyboard shortcuts make that a rapid and fluid operation, and I can keep moving the window along to further workspaces by just hitting Right (or Left) again while still holding down the other keys.

(As I've experienced many times before, having this as an easy and rapid operation encourages me to use it; I shuffle windows around this way on my laptop much more than I do on my desktops, where moving windows between virtual screens is a more involved process that generally requires multiple steps.)

Every so often I've thought about trying to create a version of this keyboard shortcut in fvwm, but so far I haven't seen a good way to do it. Although fvwm has functions and supports some logic operations, the feature's actually got a variety of challenges in fvwm's model of windows and what the current window can be. I'm pretty sure that if I looked at the actual Cinnamon code for this, it would turn out to be much more complicated than you'd expect from such a simple-sounding thing.

(I already have a keyboard shortcut for just moving to a different virtual screen; the tricky bit in fvwm is taking the current window along with me when (and only when) it's appropriate to do so given the state of the current window. I suppose the easy way to implement this is to assume that if I hit the 'take the window with me' shortcut, I've already determined that what fvwm considers the current window should be moved to the target virtual screen and my fvwm function can just ignore all of the possible weird cases.)

CinnamonWorkspaceFlipLike written at 00:23:06; Add Comment

2020-02-17

The uncertainty of an elevated load average on our Linux IMAP server

We have an IMAP server, using Dovecot on Ubuntu 18.04 and with all of its mail storage on our NFS fileservers. Because of historical decisions (cf), we've periodically had real performance issues with it; these issues have been mitigated partly through various hacks and partly through migrating the IMAP server and our NFS fileservers from 1G Ethernet to 10G (our IMAP server routinely reads very large mailboxes, and the faster that happens the better). However, the whole experience has left me with a twitch about problem indicators for our IMAP server, especially now that we have a Prometheus metrics system that can feed me lots of graphs to worry about.

For a while after we fixed up most everything (and with our old OmniOS fileservers), the IMAP server was routinely running at a load average of under 1. Since then its routine workday load average has drifted upward, so that a load average of 2 is not unusual and it's routine for it to be over 1. However, there are no obvious problems the way there used to be; 'top' doesn't show constantly busy IMAP processes, for example, indicators such as the percentage of time the system spends in iowait (which on Linux includes waiting for NFS IO) is consistently low, and our IMAP stats monitoring doesn't show any clear slow commands the way it used to. To the extent that I have IMAP performance monitoring, it only shows slow performance for looking at our test account's INBOX, not really other mailboxes.

(All user INBOXes are in our NFS /var/mail filesystem and some of them are very large, so it's a really hot spot and is kind of expected to be slower than other filesystems; there's only really so much we can do about it. Unfortunately we don't currently have Prometheus metrics from our NFS fileservers, so I can't easily tell if there's some obvious performance hotspot on that fileserver.)

All of this leaves me with two closely related mysteries. First, does this elevated load average actually matter? This might be the sign of some real IMAP performance problem that we should be trying to deal with, or it could be essentially harmless. Second, what is causing the load average to be high? Maybe we frequently have blocked processes that are waiting on IO or something else, or that are running in micro-bursts of CPU usage.

(eBPF based tracing might be able to tell us something about all of this, but eBPF tools are not really usable on Ubuntu 18.04 out of the box.)

Probably I should invest in developing some more IMAP performance measurements and also consider doing some measurements of the underlying NFS client disk IO, at least for simple operations like reading a file from a filesystem. We might not wind up with any more useful information than we already have, but at least I'd feel like I was doing something.

LoadAverageIMAPImpactQuestion written at 22:22:22; Add Comment

The case of mysterious load average spikes on our Linux login server

We have a Linux login server that is our primary server basically by default; it's the first one in numbering and the server a convenient alias is pointed to, so most people wind up using it. Naturally we monitor its OS level metrics as part of our Prometheus setup, and as part of that a graph of its load average (along with all our other interesting servers) appears on our overview Grafana dashboard. For basically as long as we've been doing this, we've noticed that this server experiences periodic and fairly drastic short term load average spikes for no clear reason.

A typical spike will take the 1-minute load average from 0.26 or so (the typical load average for it) up to 6.5 or 7 in a matter of seconds, and then immediately start dropping back down. There seems to often be some correlation with other metrics, such as user and system CPU time usage, but not necessarily a high one. We capture ps and top output periodically for reasons beyond the scope of this entry, and these captures have never shown anything in particular even when they capture the high load average itself. The spikes happen at all times, day or night and weekday or weekend, and don't seem to come in any regular pattern (such as every five minutes).

The obvious theory for what is going on is that there are a bunch of processes that have some sort of periodic wakeup where they do a very brief amount of work, and they've wound up more or less in sync with each other. When the periodic wakeup triggers, a whole bunch of processes become ready to run and so spike the load average up, but once they do run they don't do very much so the log-jam clears almost immediately (and the load average immediately drops). Since it seems to be correlated with the number of logins, this may be something in systemd's per-login process infrastructure. Since all of these logins happen over SSH, it could also partly be because we've set a ClientAliveInterval in our sshd_config so sshd likely wakes up periodically for some connections; however, I'm not clear how that would wind up in sync for a significant number of people.

I don't know how we'd go about tracking down the source of this without a lot of work, and I'm not sure there's any point in doing that work. The load spikes don't seem to be doing any harm, and I suspect there's nothing we could really do about the causes even if we identified them. I rather expect that having a lot of logins on a single Linux machine is now not a case that people care about very much.

LoadAverageMultiuserSpikes written at 01:19:38; Add Comment

2020-02-09

I'm likely giving up on trying to read Fedora package update information

Perhaps unlike most people, I apply updates to my Fedora machines through the command line, first with yum and now with dnf. As part of that, I have for a long time made a habit of trying to read the information that Fedora theoretically publishes about every package update with 'dnf updateinfo info', just in case there was a surprise lurking in there for some particular package (this has sometimes exposed issues, such as when I discovered that Fedora maintains separate package databases for each user). Sadly, I'm sort of in the process of giving up on doing that.

The overall cause is that it's clear that Fedora does not really care about this update information being accurate, usable, and accessible. This relative indifference has led to a number of specific issues with both the average contents of update information and to the process of reading it that make the whole experience both annoying and not very useful. In practice, running 'dnf updateinfo info' may not tell me about some of the actual updates that are pending, always dumps out information about updates that aren't pending for me (sometimes covering ones that have already been applied, for example for some kernel updates), and part of the time the update information itself isn't very useful and has 'fill this in' notes and so on. The result is verbose but lacking in useful information and frustrating to pick through.

The result is that 'dnf updateinfo info' has been getting less and less readable and less useful for some time. These days I skim it at best, instead of trying to read it thoroughly, and anyway there isn't much that I can do if I see something that makes me wonder. I can get most of the value from just looking at the package list in 'dnf check-update', and if I really care about update information for a specific package I see there I'm probably better off doing 'dnf updateinfo info <package>'. But still, it's a hard to let go of this; part of me feels that reading update information is part of being a responsible sysadmin (for my own personal machines).

Some of these issues are long standing ones. It's pretty clear that the updateinfo (sub)command is not a high priority in DNF as far as bug fixes and improvements go, for example. I also suspect that some of the extra packages I see listed in 'dnf updateinfo info' are due to DNF modularity (also), and I'm seeing updateinfo for (potential) updates from modules that either I don't have enabled or that 'dnf update' and friends are silently choosing to not use for whatever reasons. Alternately they are base updates that are overridden by DNF modules I have enabled; it's not clear.

(Now that I look at 'dnf module list --enabled', it seems that I have several modules enabled that are relevant to packages that updateinfo always natters about. One update that updateinfo talks about is for a different stream (libgit2 0.28, while I have the libgit2 0.27 module enabled), but others appear to be for versions that I should be updating to if things were working properly. Unfortunately I don't know how to coax DNF to show me what module streams installed packages come from, or what it's ignoring in the main Fedora updates repo because it's preferring a module version instead.)

FedoraNotReadingUpdateinfo written at 23:24:37; Add Comment


Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.