Wandering Thoughts archives

A tiny systemd convenience: it can reboot the system from RAM alone

One of the things I do a fair bit of is building and testing from-scratch system installs. Not being crazy, I do this in virtual machines (it's much faster that way). When you do this sort of work, you live in a constant cycle of installing a machine from scratch, testing it, and then damaging the install enough so that when you reboot, your VM will repeat the 'install from scratch' part. Most of the time, the most convenient way to damage the install is with dd:

dd if=/dev/zero of=/dev/sda bs=1024k count=32; sync
reboot

(The sync can be important.)

Dd'ing over the start of the (virtual) disk makes sure that there isn't a partition table and a bootloader any more, and it also generally prevents the install CD environment from sniffing around and finding too many traces of your old installed OS.

On normal System V init or Upstart based systems, this sequence has a minor little irritation: the reboot will usually fail. This is because the reboot process needs to read files off the filesystem, which you've overwritten and corrupted with the dd. Then you (I) get to go off to the VM menus and say 'power cycle the machine', which is just a tiny little interruption.

With systemd, at least in Ubuntu 16.04, this doesn't happen. Sure, a number of things run during the reboot process will spit out various errors, but systemd continues driving everything onwards anyways and will successfully reboot my virtual machine with no further activity on my part. The result is every so slightly more convenient for my peculiar usage case.

I believe that systemd can do this for several reasons. First, systemd parses and loads all unit files into memory when it starts up (or you tell it 'systemctl daemon-reload'), which means that it doesn't have to read anything from disk in order to know what needs to be done to shut the system down. Second, systemd mostly terminates processes itself; it doesn't need to repeatedly get scripts to run kill and the like, which could fail if kill or other necessary bits have been damaged by that dd. Finally, I think that systemd can handle calling reboot() internally, instead of having to run an executable (which might not be there) in order to do this.

(Systemd clearly has internal support in PID 1 for rebooting the system under some circumstances. I'm not quite clear if this is the path that a normal reboot eventually takes; it's a bit tangled up in units and handling this and that and so on.)

PS: Possibly there is a better way to damage a system this way than dd. dd has the (current) virtue of being easy to remember and clearly sufficient. And small variants of this dd command work on any Unix, not just Linux (or a particular Linux).

ZFS on Linux has just fixed a long standing little annoyance

I've now been running ZFS on Linux for a while. Over that time, one of the small little annoyances of the ZoL experience has been that all ZFS commands required you to be root, even if all you wanted to do was something innocuous like 'zpool status' or 'zfs list'. This wasn't for any particularly good reason and it's not how Solaris and Illumos behave; it was just necessary because the ZoL kernel code itself had no permissions restrictions on anything for complicated porting reasons. Anyone who could talk to /dev/zfs could do any ZFS operation, including dangerous and destructive ones, so it had to be restricted to root.

Like many people running ZoL, I dealt with this in a straightforward way. To wit, I set up a /etc/sudoers.d/02-zfs file that allowed no-password access to a great big list of ZFS commands that are unprivileged on Solaris, and then I got used to typing things like 'sudo zpool status'. But this was never a really great experience and it's always been a niggling annoyance.

I'm happy to report that as of a week or so ago, the latest development version of ZoL now has fixed this issue. Normal non-root users can now run all of the ZFS commands that are unprivileged on Solaris. As part of this, ZoL now supports normal ZFS 'zfs allow' and 'zfs unallow' for most operations, so you can (if desired) allow yourself or other normal users to do things like create snapshots.

(Interestingly, poking around at this caused me to re-discover that 'zpool history' is a privileged operation even on Solaris. I guess some bits of my sudoers file are going to stay.)

Things like this are part of why I've been pretty happy to run the development version of ZoL. Even the development version has been pretty stable, and it means that I've gotten a fair number of interesting and nice features well before they made it into one of the infrequent ZoL releases. I don't know how many people run the development version, but my impression is that it's not uncommon.

(I can't blame the ZoL people for the infrequent releases, because they want releases to be high quality. Making high quality releases is a bunch of work and takes careful testing. Plus sometimes the development tree has known outstanding issues that people want to fix before a release. (I won't point you at the ZoL Github issues to see this, because there's a fair amount of noise in them.))

Some notes on how xdg-open picks which web browser to use

Let's start with my tweet:

It sure would be convenient if xdg-open documented how it picked which browser to use, so you could maybe change that to one you like.

XDG here is what used to be called the 'X Desktop Group' and is now freedesktop.org (per wikipedia). As you might then guess, xdg-open the program is the canonical KDE/GNOME/etc agnostic way of opening (among other things) an URL in whatever is nominally your preferred browser. Since providing a web interface as your package's UI is an increasingly common thing to do these days, there are any number of programs and scripts and so on that want to do this and do it by just running xdg-open, which gives you whatever strange browser xdg-open decided to inflict on you today.

If you read the xdg-open manpage, it's remarkably uninformative about how it decides what your default or preferred web browser is. On the one hand xdg-open is currently a shell script so you could just read it to find this out; on the other hand, it's a 900+ line shell script full of stuff, which makes it not exactly clear 'documentation'. So here are some notes on the current behavior, boiled down to the useful bits.

(Because none of this is documented, freedesktop.org is free to change it at any time if they feel like it. They probably won't, though.)

Broadly, xdg-open does the following when you ask it to open an URL:

• first, it tries to figure out if you're running a desktop environment that it knows about. If you are, it uses the desktop's own mechanism to open URLs if there is one (these are things like kde-open, gvfs-open, and exo-open). The important thing here is that this should automatically respect your normal desktop browser preference, since it's using the same mechanism.

• if $BROWSER is set, it's used immediately. I'll discuss what it can be set to later.

• in some but not all versions of xdg-open, it will attempt to determine the correct program to handle the nominal MIME type of 'x-scheme-handler/http' (or https, or ftp, or whatever) and then run this program. This determination is made by xdg-mime; how it works is complicated enough to require another entry.

  In versions of xdg-open that support this, it's extremely likely that this will produce a result and xdg-open will never go on to its final option.

• if all else fails, xdg-open tries to run a whole bunch of possible browsers (via the $BROWSER mechanism). The most important thing about this is that the first default browser xdg-open tries is x-www-browser.

Some but not all Linux distributions have a /usr/bin/x-www-browser. If yours has one, it may or may not point to something useful or what you want, and it may vary from version to version (and on what packages you wind up installing). For example, on our Ubuntu 12.04 and 14.04 machines x-www-browser points to arora by default, which is not what I consider a particularly good choice on machines with Firefox installed (it's certainly likely to puzzle users who wind up running it). On 16.04 it points to Chromium, which is at least a plausible default choice.

On current Fedora and Ubuntu 16.04, the full list of (graphical) browsers, in order, is:

x-www-browser firefox iceweasel seamonkey mozilla epiphany konqueror chromium-browser google-chrome

The useful thing here is that xdg-open searches for all of these using your $PATH, so you can create a personal x-www-browser symlink that points to whatever you want without having to risk unintended side effects from eg setting $BROWSER.

If your xdg-open uses xdg-mime, you can determine what program it will use (more or less) by running eg:

$ xdg-mime query default x-scheme-handler/http
firefox.desktop

Xdg-mime will let you set this stuff as well as query it, but of course doing so may have more consequences than just changing xdg-open's behavior. However at that point there is no really good solution; if xdg-open is consulting xdg-mime, I think all you can do is either set $BROWSER or set xdg-mime's defaults.

(If you're setting scheme handlers in xdg-mime, remember to get at least http and https, and perhaps ftp as well.)

While the xdg-open manpage will refer you to the xdg-settings manpage, this is potentially quite misleading. If you aren't using a desktop environment, there's no guarantee that what xdg-settings reports as your default browser is what xdg-open will necessarily run. The two scripts use different code and as far as I can tell, neither makes any particular attempt to carefully duplicate each other's logic.

(If you are running a desktop environment that xdg-settings knows about, well, you're at the mercy of how well it can read out your DE's default browser. I wouldn't hold my breath there.)

Sidebar: What $BROWSER can be set to

As xdg-open interprets it, $BROWSER is a colon separated list of commands, not just program names (ie you can include things like command line switches). A command may contain a %s, which will be substituted with the URL to be opened; otherwise the URL is just tacked on the end of the command. Xdg-open does not exactly have sophisticated handling of quoting and argument splitting or anything, so I would do nothing tricky here; I wouldn't even really trust the %s substitution. If you think you need it, use a cover shell script.

How xdg-mime searches for MIME type handlers (more or less)

XDG is what was once called the X Desktop Group and is now freedesktop.org. Part of their work is to create various cross-desktop and desktop agnostic standards and tools for how to do things like determine what program should handle a particular MIME type. This includes special 'scheme' MIME types, like x-scheme-handler/ftp, which means 'whatever should handle FTP' (technically this is 'FTP urls'). The XDG tool for mapping MIME types to programs (actually .desktop files) is xdg-mime. Like all of the XDG tools, xdg-mime uses a collection of environment variables, which will normally have the default values covered in the XDG base directory specification.

In theory there are two sorts of data that xdg-mime looks at. The first, found in files called <desktop>-mimeapps.list and mimeapps.list, is supposed to be a set of specifically chosen applications (whether configured by the user or the system). The second is a general cache of MIME type mapping information based on what program can handle what MIME types (as listed in each program's .desktop file); these are found in files called defaults.list and mimeinfo.cache. In practice, a system mimeapps.list file may have just been slapped together based on what the distribution thought was a good idea, and it may not correspond to what you want (or even what you have installed).

(Not all Linux distributions ship a system mimeapps.list. Fedora does; Ubuntu doesn't.)

Xdg-mime searches for mimeapps.list in any of $XDG_CONFIG_HOME, $XDG_CONFIG_DIRS, and the 'applications' subdirectories of $XDG_DATA_HOME and $XDG_DATA_DIRS. It searches for the other two files only in the applications subdirectories of $XDG_DATA_HOME and $XDG_DATA_DIRS.

Now, we get to an important consequence of this. $XDG_DATA_DIRS is normally two directories, /usr/local/share and /usr/share, each of which has MIME type information only for itself, and they are checked in order. The normal result of this is that things installed into /usr/local steal MIME types from things installed in /usr because xdg-mime will check the /usr/local/share/applications files first.

(I discovered this when I installed Google Chrome on my laptop and it promptly stole URL handling from Firefox, which I did not want, because the RPM put bits into /usr/local instead of /usr. Actually finding the files that controlled this was remarkably hard.)

Normally, nothing automatically generates or updates the system mimeapps.list; on Fedora, it's part of the shared-mime-info RPM (and kde-settings for the kde-mimeapps.list version). The mimeinfo.cache files are maintained by update-desktop-database, which will normally be automatically run on package installs, removals, and probably upgrades.

Now, xdg-mime does not give you an actual program to run. What it does is give you the name of a .desktop file, eg firefox.desktop. In order to use this to run a program, you must look through the right magic places to find the .desktop file and then parse it to determine the command line to run. Probably you don't want to do this yourself, but I don't know what your alternative is; as far as I know, there is no XDG tool script to say 'run this .desktop command with the following arguments'.

(Note that the .desktop file is not necessarily in the same directory as the MIME mapping file that caused it to be the handler application. For example, your $HOME/.local/share/applications might just have various MIME type overrides you've set for what system application should handle what.)

This explanation is somewhat dependent on what exact version of the XDG tools and scripts you have installed. It's also not necessarily totally complete, because I am reading through undocumented shell scripts here and I've left a few things out. See the Arch wiki entry on default applications for much more information.

PS: If you're trying to track down why xdg-mime is giving you some strange result, set $XDG_UTILS_DEBUG_LEVEL to at least '2'. This will tell you just what files it looked at when, although I don't think it ever reports what directories it looked at but didn't find any files in.