Wandering Thoughts: Recent Entries

Choosing the superblock format for Linux's software RAID

Linux's software RAID implementation stores metadata about the RAID device in each physical device involved in the RAID, in what mdadm calls 'RAID superblocks' by analogy to the filesystem superblocks that describe filesystems. In modern versions of software RAID there are a number of different formats for these RAID superblocks with different tradeoffs involved in each one, and one of the decisions you need to make when you create a software RAID array is what format you want to use.

(Even if you don't actively make a decision, mdadm will pick a format for you. Sometimes it will whine irritatingly at you about the situation, which is how I discovered the whole issue.)

In my opinion, at the moment there are three sensible options to choose from: the 0.90 format and then two variants of the 'version-1' metadata format.

• 0.90 is the original metadata format, which is widely understood and used. For most people, the most potentially important limitation of 0.90 metadata is that component devices can't be larger than 2 TB.

  The 0.90 superblock goes at the end of the underlying partition.

• 1.0 puts the superblock at the end of the underlying partition.
• 1.2 puts the superblock 4 Kb from the start of the underlying partition It's the sort of default for modern versions of mdadm.

(You can see what format your current RAID arrays are using by looking at /proc/mdstat. If an array doesn't say 'super <something>' it's using 0.90 format metadata; otherwise, it's using whatever version it says it is. Many relatively modern systems, such as Ubuntu 10.04, either don't support anything past 0.90 or default to 0.90 in system setup.)

Where the superblock goes is potentially important for RAID-1 arrays. A RAID-1 array with the superblock at the end can relatively easily have whatever filesystem it contains mounted read-only without the RAID running, because the filesystem will start at the start of the underlying raw partitions; this can be important sometimes. A RAID-1 array with the superblock at or near the start of the underlying partitions can't have the raw partitions used this way, because you have to look somewhat beyond the start of the raw partition to see the filesystem.

(Some versions of mdadm will explicitly warn you about this or even quiz you about it if you don't specify a format explicitly.)

If you want to use a modern format and are going to directly use the RAID-1 array for a filesystem, I would use 1.0 format (this is what I've done for my new / and /boot). For swap areas you might as well use 1.2 format; if you ever need to use swap without software RAID, you can just destroy the 1.2 superblocks with mkswap. For LVM physical volumes you can argue back and forth either way; right now I've chosen 1.2 format because I really don't want to think about what it would take to safely bring up an LVM physical volume without software RAID running.

(LVM physical volumes have their own metadata, which normally goes at the start of the 'raw' partition that LVM is using but which can be replicated to the end as well. See pvcreate's manpage.)

As far as I know you can't change the superblock format of an array after it has been created, at least not without destroying it and recreating it. You can sort of do this without an extra disk with sufficient work, but really you want to get it right at creation time.

PS: note that in theory you can use dmsetup to gain access to filesystems or other sorts of data that doesn't begin at the start of a raw partition, so you can get at a filesystem embedded inside the raw partition of a RAID-1 array with 1.2 format metadata. However this requires user level intervention, which means that you're going to need a rescue environment or rescue disk of some sort.

The solution to the modern X font handling mystery

I wrote last time about my attempts to work out just why xterm was rendering the same font differently on Ubuntu and Fedora. Thanks to comments from Adam Sampson and some additional digging, I now have an answer and some theories. As it happens, the answer illuminates yet more issues with modern X font handling.

In the modern Xft/FreeType/Fontconfig world, fonts are specified more or less as a font name and a size. With most programs that allow explicit specification of the font name you can augment the name with additional attributes, partly to modify the exact font that gets matched and partly to control how it's rendered. All of this is sort of covered in the fontconfig user documentation.

(An example could be 'DejaVu Sans Mono:style=bold:hintstyle=hintslight'. This shows both a modification of the font selection process and a rendering instruction. A similar sort of syntax can be used if you want to find, eg, all of the monospace fonts on the system.)

Fontconfig also has system-wide configuration files, found in /etc/fonts/conf.d/. In most packages that I'm familiar with, the global configuration is a default and explicit specification of things override them. However, this is not the case for fontconfig; at least for some settings, fontconfig's global settings silently override anything you specify explicitly. The only way to override these settings yourself is to have a $HOME/.fonts.conf file (and you can't unset the settings so that you can pick them on the fly, only set them to whatever personal global value you want).

You can probably guess the rest of the story. As spotted by Adam Sampson, Ubuntu's fontconfig package has a global config file that is explicitly forces hinting to be set to hintslight, while Fedora has no config file and is defaulting to hintfull. Because this is set in a global config file you can't override it on the xterm command line, which fooled me into thinking that this setting wasn't the culprit.

(You can include ':hintstyle=hint<whatever>' in a -fa argument all you want, but it is silently ignored.)

Overriding that (with a personal .fonts.conf file that forces hintfull hinting) got Ubuntu rendering to be almost the same as Fedora rendering. The remaining difference turns out to be due to the specific versions and compilation options of my version of FreeType. Interestingly, this is not just a small visual difference; at least under some circumstances the Ubuntu FreeType library renders DejaVu Sans Mono characters a pixel or so taller than my Fedora FreeType library does, meaning that an 80x50 xterm on Ubuntu is visibly taller than a Fedora 80x50 xterm. (They are both the same width.)

I don't know for sure why gnome-terminal, Firefox, and TK applications were unaffected by this, but my theory is that all of them use the Gnome preferences system. Gnome has its own preferences settings for how to render fonts and these appear to completely override fontconfig's views on the subject, so Gnome applications were using the 'right' hinting style for my tastes. I would have probably seen the same rendering of DejaVu Sans Mono in any other Gnome application that used it as the monospace font (a good example is probably gedit).

(Why this happened for some fonts and not for others presumably has to do with how the fonts were hinted, or maybe some fonts specify that they can only be hinted at some levels. I don't know if this means that the fonts that weren't affected are less hinted than DejaVu Sans Mono and so on, or just hinted differently.)

The drawback of modern X font handling gets mysterious

Back in The drawback of modern X font handling I covered how modern X font rendering happens in the client and so can vary from client to client, going from nice on one client to bad on another. I illustrated this with xterm on Fedora and Ubuntu displaying the same font, Fedora well and Ubuntu badly. I now have a good reason to change to using xterm with modern fonts, so I spent part of today poking at this issue; the results have turned this into a genuine peculiar mystery.

What I have so far:

• the problem does not happen with all programs on Ubuntu. So far xterm and GNU Emacs have the bad font rendering, but Firefox, gnome-terminal, and TK-based programs such as exmh and tkmsg do not; they render DejaVu Sans Mono just like Fedora does.

• the problem only happens with some monospace fonts, not all of them. The Ubuntu machine I was testing on has 11 candidate fonts listed by 'fc-list :scalable=true:spacing=mono: family'; seven of them show the problem but four do not.

  (The good four are TlwgMono, Tlwg Typo, Courier New, and FreeMono. Unfortunately my preferred xterm font is DejaVu Sans Mono.)

• the problem is not the Ubuntu version of xterm, the Ubuntu app-defaults file for xterm, or even the Ubuntu Freetype library; I have built the Fedora xterm and my version of Freetype on Ubuntu and used the Fedora app-defaults, and the bad rendering is still there.

• I've directly set several fontconfig font rendering options that might be doing this without changing anything; at this point I haven't seen any difference with autohint, weight, embolden, or aspect (the last was a wild shot). Similarly, Xft X resources (cf) do nothing that I can see.

  (Forcing autohint=true actually makes the Fedora font rendering slightly but visibly darker while leaving the Ubuntu rendering unchanged for both the good and bad programs.)

• the problem doesn't happen with xterm on some FreeBSD machines I have handy; they render DejaVu Sans Mono the good way.

Clearly something mysterious is happening in the depths of the Ubuntu version of Xft or something it calls, but only if it's invoked in the right (or wrong) way. Unfortunately I don't think there's any good way for non-experts to see what font rendering choices are being made (the fontconfig library can be coaxed into some debugging output, but it's pretty much 'exports only' from what I can see), so I have no idea if I'll be able to figure out a solution that lets me use the font I want.

(Changing to gnome-terminal is not a solution for me.)

What you can find out about the memory usage of your Linux programs

Recently I wound up reading Linux Memory Reporting (via Hacker News), where Robert Haas talks about Linux's lack of clear reporting on process memory use. Today I'm going to sort of answer his question by covering what information Linux gives you about the various sorts of memory usage that you could be curious about. My primary focus is going to be on the numbers that you can get with ps, top, and smem. The background information on general Unix memory management will be helpful.

So, what you can get:

• the total amount of virtual address space that your process currently has allocated and mapped is the 'virtual size' of your process; ps reports this as VSZ and top reports it as VIRT. This includes anything the process has mapped, regardless of how it got there; the program's code, shared libraries, (System V) shared memory areas, mmap()'d files, mmap()'d private anonymous memory areas (which are often used by the C library for malloc()), everything.

  If your program is in a steady state but your VSZ keeps increasing, you have some sort of allocation leak. It may not strictly be a memory leak; you might be forgetting to unmap files or unload dynamically loaded code or something.

  (You can check at least some of this with lsof.)

• how much RAM would be immediately freed up if this process exited is smem's USS ('unique set size') field; this counts pages of RAM that the process is the only user of. These pages may be private pages (pages that will never be accessible by anyone else), or they may be shared pages that are only actively used by this process.

  (smem gets this information from the per-process smaps proc file.)

• how much RAM your program has looked at recently (which is roughly how much RAM it needs to be happy if it wasn't sharing anything) is the 'resident set size', reported as RSS by ps and smem and RES by top. The resident set size doesn't care whether or not some of that RAM is also used by other processes; each process counts it up separately.

  (In the terminology of my basic Unix memory management entry, a process's RSS is just how many page table entries in its virtual memory areas point to real RAM.)

  Your process's RSS increases every time it looks at a new piece of memory (and thereby establishes a page table entry for it). It decreases as the kernel removes PTEs that haven't been used sufficiently recently; how fast this happens depends on how much memory pressure the overall system is under. The more memory pressure, the more the kernel tries to steal pages from processes and decrease their RSS.

  If you have a memory leak it's routine for your RSS to stay constant while your VSZ grows. After all, you aren't looking at that leaked memory any more.

  A large RSS on an active system (one under memory pressure) means that your process touches a lot of memory (often rapidly) during its operation. A growing RSS means that it is increasing the amount of memory it touches. A constant RSS doesn't mean that the process is touching the same memory over and over; it just means that it's touching about the same amount of memory per unit time.

• the process's fair share of currently in use RAM is smem's PSS ('proportional set size') field. This prorates shared pages of RAM by charging each process for 1/Nth of the page, where N is how many processes currently have a page table entry for the page (the degenerate case is that you are charged the full page if you are the only user, ie this would be counted as part of your USS). Note that this is not how many processes have the shared resource mapped into their address space, it is how many processes have touched the page recently (ie, have it in their RSS). Mapping a shared resource is free (except to your VSZ); looking at it is what costs you here.

  It follows that the more processes actively look at pages of a shared resource, the lower each of their PSS goes for it (because more and more processes map the same pages from it).

  (Like USS, smem gets this information from the per-process smaps proc file.)

Because of how it's defined, summing the per-process PSS for a resource over all of the processes using that resource will tell you how much RAM that resource is using. Smem can do this (for some resources) with 'smem -m', although you need to know a certain amount about how Linux gives names to various resources in order to understand smem's output here.

(If you have all of the processes of interest running under a single userid, you can also use 'smem -u'. Smem doesn't currently have an option to aggregate reporting by program, so you can't do things like see how much memory your httpd processes are collectively using.)

As far as I know, Linux has no per-process or global number for how much of your virtual address size has ever been looked at (my second question in the six different meanings of memory usage). Nor can you get per-process information on how much memory the operating system might need to provide if your process wrote to everything it was entitled to (the sixth question), although you can get system-wide information on committed address space.

Top reports a SHR number but it's not clear to me how useful this is, partly because top doesn't document where it gets this information from. If I am reading the kernel code correctly, the most likely source is the (process) RSS for memory areas that were mmap()'d from files. I am not sure if this includes things like System V shared memory areas, and certainly it understates the potential sharing between, say, fork()'d processes. This is also only potential sharing, since it says nothing about whether or not any other process has mmap()'d the same object.

(Ie, if your single process mmap()'s a private two gigabyte file and then scans all of it, I believe that your SHR will be two gigabytes and change.)

Sidebar: answers to Bruce Momjian's questions

From his comment on Robert Haas's entry:

There are various methods for representing memory that is shared, either via SysV shared memory, fork's copy-on-write, or shared libraries. Does every process get charged the full amount, or do they split it among themselves, e.g. if five processes use shared memory, is each process charged 20% of the total size? (If another process attaches, does your percentage decrease?) What happens when you map in a large shared memory area but only access part of it? When do you stop using that memory?

Each process is charged the full amount to VSZ, but not to other numbers. When you map a large area but only refer to some of it, your VSZ goes up by the full amount but your RSS only goes up by the amount you access (and then goes down again at some rate if you don't access it and the system is under memory pressure). Your PSS is the only number that goes down if other people attach to the shared resource and also actually look at pages of that shared resource that you are also looking at (if they attach but don't look, your PSS doesn't change). If five processes all map the same shared memory segment but look at five different portions of it, each of them will be charged separately for their portion (their PSS for the segment will be the same as their USS); if they all look at the same portion, their PSS is 1/5th of the size of the portion.

(Your RSS never changes when people attach or detach from a shared resource.)

Notes on what Linux's /proc/<pid>/smaps fields mean

Because I was just digging around in the kernel source to determine this (it's a long story), here is some notes about what the fields of the smaps file mean and how they're calculated. The factory for this particular sausage is fs/proc/task_mmu.c (at least as of the current git tree).

For each VMA mapping that gets listed in smaps, the kernel walks all of the PTEs associated with it and looks at all of the known pages. Each PTE is then counted up:

• the full PTE size is counted as Rss.
• if the page has been used recently, it's added to Referenced.
• if the page is mapped in only one process it is labeled as private; its full size is added to Pss.
• if the page is mapped in more than one process it is shared and the amount it adds to Pss is divided by the number of processes that have it mapped.

(If the PTE is for something in swap it only adds to the Swap size.)

Note that a 'private' page is not quite as private as you might think. Because processes map pages independently of each other, it's possible to have a shared page that is currently mapped only by a single process (eg only one process may have called an obscure libc function recently); such pages are counted in 'private'.

The Size of a mapping is how much address space it covers.

If the mapping has been locked into memory via mlock() or the like, Locked is the same as Pss (ie, it is this process's fair share of the amount of locked memory for this mapping); otherwise it is 0 kB.

Given that looking at smaps requires walking the pages of all of the VMAs, I suspect that it's a reasonably costly operation. It'd probably be a bad idea to build a tool that did it a lot, especially if the tool scanned all processes in the system.

(Smem uses smaps, but it doesn't normally run repeatedly in the way that, say, top does.)

A Yum plugin I would like: using a local DVD as a repo source

It's become obvious to me that, to put it one way, Fedora 16 is where all of the update action is and Fedora 15 is not getting many changed packages (this is probably well known among people who actually pay attention to how Fedora is structured). This means that I really need to upgrade my home machine from Fedora 15 to Fedora 16. Because I'm sane I'll be doing this with a yum upgrade, which means that I need to get several gigabytes of all of those RPMs.

On my work machine, this is no particular problem because I have fast networking; I'm not going to notice fetching even gigabytes of data (and it goes basically as fast as the other end can feed it to me). At home, well, not entirely so much; I have a much better DSL downlink than I used to, but it is not really all that fast and I will definitely notice if it's in use.

What I would like to be able to do is use a local DVD as the source of as many of those packages as possible. The obvious DVD to use is the normal Fedora install DVD (hopefully most of the packages I need will be coming from the base Fedora 16 repository anyways instead of from the Fedora 16 updates). A plugin to do this would be useful for more than yum upgrades; among other things, you could also use it to easily add more packages after the upgrade (or after a from-scratch install from the DVD).

(This plugin would even be useful at work. Even though I do yum upgrades at work I often download the Fedora DVD image so that I can test in VMWare and do other things with it. It seems silly to download the same packages twice, once in a DVD image and once for a yum upgrade.)

While I think you can do this with a carefully created repos.d file, the plugin I'd really like would automatically notice things that look like mounted DVDs, check them for an install-DVD-like structure, figure out what Fedora version they're for, and create a repo on the fly as appropriate. This is probably a pipedream.

Using 'yum --downloadonly' overnight sidesteps a lot of the bandwidth issues for my specific case but it still feels like a wasted opportunity.

(I'd also like to be able to use a local DVD as a package source for mock's build environments. Repeatedly downloading base package sets every so often is not a really good use of my DSL link.)

PS: it's possible that what I want already exists and I just haven't found it. I think mock has some support for this, which I haven't investigated extensively as I don't often need to use mock at home.

How to use systemd to just run some stuff on boot

Suppose that you have some things that you want to get run when your system boots, much like rc.local. They aren't necessarily daemons, you don't want to wire them up to the whole systemd magic infrastructure, you just want to run something. Let us assume that you have some shell scripts for simplicity (if you don't, it's easy to convert what you need into one or more shell scripts).

When I was converting my old init.d stuff to systemd, here is how I did this:

[Unit]
Description=Run my stuff
After=network.target
Requires=network.target

[Service]
Type=oneshot
RemainAfterExit=True
ExecStart=/some/script
ExecStart=/some/other/script

[Install]
WantedBy=multi-user.target

(The systemd service that actually runs /etc/rc.d/rc.local is a bit different; see /lib/systemd/system/rc-local.service, at least on Fedora.)

If you want your rc.local equivalent to be started just as gettys and any graphical login manager are being started, it appears that you want to be after systemd-user-sessions.service. Most of the startup stuff I do doesn't need to be run that late but it does depend on networking being fully up so that the machine has an IP address and all its interfaces and so on.

(One of my wishlist items for systemd is the ability for services to depend on and be triggered on various sorts of network state changes. I suspect that the systemd people see this as more the job of NetworkManager and dbus in general, but NM is not something that I can use and I'd rather avoid having a second dependency and state change management system to handle dbus events when we already have a perfectly good general one in systemd.)

Nailing down RPM epoch numbers

As I mentioned here, RPM package version numbers have three components: an 'epoch' number, the package's upstream version, and an RPM package release version. The epoch number is there because upstream developers sometimes change version numbers in ways that break RPM's normal version comparison algorithm, or just plain use odd version formats.

(For example, I believe there's at least one program that increments its version by adding another digit of π to the end of the version number. This makes perfect sense in a way, but is hard for RPM version comparison to cope with.)

In that other entry I wrote that the default epoch number was '1' (and then a commentator corrected me to say that it was '0'). Both of us turn out to be wrong. The default RPM versioning is to have no epoch number at all. A RPM package that does not specify an epoch number does not have an epoch of '0'; it has no epoch at all, which is displayed by RPM as '(none)'. A package with no epoch number is a lower version overall than any epoch number, even 0.

On all of the RPM-based machines that I have sitting around, no epoch is by far the most common epoch. The next most common epoch is actually '1', not '0'; one theory I have about this is that a lot of people who build RPMs also think that the default epoch is 0, so when they specify an explicit epoch to override the upstream version number they start from 1.

Across all of the packages that I have handy on my machines, the highest epoch number is 50 (on aspell-en, on Fedora 14, Fedora 15, and Fedora 16). Other high epochs (on Fedora 16) are 32 (for bind's packages), 14 (tcpdump plus libpcap), and 12 (dhcp and aspell).

If you want to check your own RPM-based systems, appropriate rpm commands are:

rpm -qa --qf '%{E}\n' | sort | uniq -c | sort -nr
rpm -qa --qf '%{E}:%{N}\n'

The first will show you a count of epoch numbers; the second will show you the epochs for all of your packages (so you can find, for example, the package or packages at epoch 50).

How not to improve your package updater application

Earlier today I tweeted:

Every time I use the Fedora 16 gpk-update-viewer I'm filled with anger for what someone did to a nice, usable program.

Now I feel like explaining that (in the spirit of an earlier entry). First, gpk-update-viewer is the program behind the 'Software Update' information display on Fedora systems; it's part of PackageKit.

In Fedora 15, the (Gnome) interface for it has two relatively minor issues (which you can actually see in the current screenshots on the PackageKit website): it doesn't group the updates into separate sections for security updates, bugfix updates, and regular updates, and it always shows the 'Details' area even if you're not interested in it. In the Fedora 16 version, someone decided to fix these problems. To properly appreciate the result, I must show you a picture (which I am not going to inline here because it's too big): a screenshot of the Fedora 16 gpk-update-viewer.

The Fedora 16 version has certainly fixed those two little problems; the Details area can now be folded away, and updates are now grouped. However this 'improvement' has created all of the following issues, many of them disastrous:

• the 'Details' area for the details of the updates is not resizable; it's fixed at four and a bit lines, which is completely and utterly inadequate for usefully reading those details without endless scrolling. I am now very glad that I recently contrived a scroll wheel.
• if you scroll the 'Details' area (and you will be) and then move to another package, the area's scroll position is not reset to the top. It doesn't always stay static either; at least sometimes it shifts oddly.

Since the entire reason I'm running gpk-update-viewer at all is to read the update details, this change alone is a huge usability loss. (I check for and actually apply updates through yum, but as far as I know g-u-v is the only tool for seeing the update text (which is not the same as the RPM changelogs).)

• there is no longer any indication of if the update will require a system reboot or for you to log out and back in again. Fedora 15 annotated such updates with little icons.

• Fedora 15 mostly grouped all related binary packages together as a single entry (which you could expand to see the list of individual packages that would be updated). Fedora 16 lists every binary package separately, even in cases where an update to a single source package produces a shower of fifteen or twenty updated binary packages.

  (You can see this in the screenshot; in Fedora 15, there would not be separate entries for 'cheese' and 'cheese-libs'. Fedora 15's version of this was incomplete, but it was still much better than not having this at all.)

• Those headings for the type of updates you see in the screenshot ('Security updates' and 'Bug fix updates') are actually selectable entries. You can click on them and, more importantly, if you are scrolling through the list of updates you will wind up on those headings too (and have to explicitly scroll past them). This is especially irritating while you are using the arrow keys to scroll to the next entry because your mouse cursor is in the Details area so you can use your scroll wheel to read the update details.

And on a slightly more minor gripe, the Fedora 16 version always starts up with a completely crazy window size when I run it under fvwm in my normal configuration. No other Gnome program seems to have this problem and the Fedora 15 version was (and is) fine.

(Fedora 15 has gnome-packagekit 3.0.0; Fedora 16 has 3.2.1. Sadly the Fedora 15 binary doesn't work on Fedora 16, but maybe I can recompile the old version from source.)

A quiet advantage of the systemd approach to service management

One of the ways that things like systemd and upstart are different from the traditional /etc/init.d approach to starting and restarting services is in who starts and restarts services. In the traditional approach, you do this directly by running /etc/init.d scripts (either explicitly or through some cover scripts). In the systemd world, things are indirect; 'systemctl start foo.service' just asks the master systemd to run the appropriate commands to start the service.

There is a somewhat subtle advantage to having systemd do this, one that I have been quietly appreciating lately: your shell environment never leaks into restarted services.

In the init.d world, a manually started or restarted service inherits whatever environment your shell had when you ran the script(s), especially your environment variables. This is different than init's environment and can quite possibly contain settings that will change how the service operates.

(Even logging in directly as root or using 'su -' doesn't necessarily make this concern go away. As we've seen before, locale settings can drastically change the behavior of programs and ssh will helpfully propagate your current ones to the remote end.)

The systemd approach makes all of those concerns go away. The environment you run systemctl or initctl in doesn't matter, because all these commands do is ask the master process to do things. The master process doesn't inherit anything from you and so the commands it runs are uncontaminated by whatever weird environment you run stuff in. Instead they're guaranteed to always be running in the same environment regardless of whether they were started at boot time or stopped and restarted later.

This issue is somewhat near and dear to my heart because I retain a fairly customized environment even when I su to root. On init.d systems, I need to go out of my way to make sure that things like mailers and sshd will inherit as few peculiar things as possible (and I'm sure I'm not always successful). On systemd systems, well, I can forget about the whole concern. I enjoy that.

(There are sometimes drawbacks to this approach; for example, you can't quickly increase the number of open file descriptors a daemon can have by changing your root shell's ulimit and then restarting the daemon, or set special debugging flags in the environment. I haven't found these drawbacks to be an issue in practice so far.)

(This entry was brought to you by me spending a bunch of time today going 'systemctl restart ...' to test some things.)