Wandering Thoughts archives

The solution to the modern X font handling mystery

I wrote last time about my attempts to work out just why xterm was rendering the same font differently on Ubuntu and Fedora. Thanks to comments from Adam Sampson and some additional digging, I now have an answer and some theories. As it happens, the answer illuminates yet more issues with modern X font handling.

In the modern Xft/FreeType/Fontconfig world, fonts are specified more or less as a font name and a size. With most programs that allow explicit specification of the font name you can augment the name with additional attributes, partly to modify the exact font that gets matched and partly to control how it's rendered. All of this is sort of covered in the fontconfig user documentation.

(An example could be 'DejaVu Sans Mono:style=bold:hintstyle=hintslight'. This shows both a modification of the font selection process and a rendering instruction. A similar sort of syntax can be used if you want to find, eg, all of the monospace fonts on the system.)

Fontconfig also has system-wide configuration files, found in /etc/fonts/conf.d/. In most packages that I'm familiar with, the global configuration is a default and explicit specification of things override them. However, this is not the case for fontconfig; at least for some settings, fontconfig's global settings silently override anything you specify explicitly. The only way to override these settings yourself is to have a $HOME/.fonts.conf file (and you can't unset the settings so that you can pick them on the fly, only set them to whatever personal global value you want).

You can probably guess the rest of the story. As spotted by Adam Sampson, Ubuntu's fontconfig package has a global config file that is explicitly forces hinting to be set to hintslight, while Fedora has no config file and is defaulting to hintfull. Because this is set in a global config file you can't override it on the xterm command line, which fooled me into thinking that this setting wasn't the culprit.

(You can include ':hintstyle=hint<whatever>' in a -fa argument all you want, but it is silently ignored.)

Overriding that (with a personal .fonts.conf file that forces hintfull hinting) got Ubuntu rendering to be almost the same as Fedora rendering. The remaining difference turns out to be due to the specific versions and compilation options of my version of FreeType. Interestingly, this is not just a small visual difference; at least under some circumstances the Ubuntu FreeType library renders DejaVu Sans Mono characters a pixel or so taller than my Fedora FreeType library does, meaning that an 80x50 xterm on Ubuntu is visibly taller than a Fedora 80x50 xterm. (They are both the same width.)

I don't know for sure why gnome-terminal, Firefox, and TK applications were unaffected by this, but my theory is that all of them use the Gnome preferences system. Gnome has its own preferences settings for how to render fonts and these appear to completely override fontconfig's views on the subject, so Gnome applications were using the 'right' hinting style for my tastes. I would have probably seen the same rendering of DejaVu Sans Mono in any other Gnome application that used it as the monospace font (a good example is probably gedit).

(Why this happened for some fonts and not for others presumably has to do with how the fonts were hinted, or maybe some fonts specify that they can only be hinted at some levels. I don't know if this means that the fonts that weren't affected are less hinted than DejaVu Sans Mono and so on, or just hinted differently.)

The drawback of modern X font handling gets mysterious

Back in The drawback of modern X font handling I covered how modern X font rendering happens in the client and so can vary from client to client, going from nice on one client to bad on another. I illustrated this with xterm on Fedora and Ubuntu displaying the same font, Fedora well and Ubuntu badly. I now have a good reason to change to using xterm with modern fonts, so I spent part of today poking at this issue; the results have turned this into a genuine peculiar mystery.

What I have so far:

• the problem does not happen with all programs on Ubuntu. So far xterm and GNU Emacs have the bad font rendering, but Firefox, gnome-terminal, and TK-based programs such as exmh and tkmsg do not; they render DejaVu Sans Mono just like Fedora does.

• the problem only happens with some monospace fonts, not all of them. The Ubuntu machine I was testing on has 11 candidate fonts listed by 'fc-list :scalable=true:spacing=mono: family'; seven of them show the problem but four do not.

  (The good four are TlwgMono, Tlwg Typo, Courier New, and FreeMono. Unfortunately my preferred xterm font is DejaVu Sans Mono.)

• the problem is not the Ubuntu version of xterm, the Ubuntu app-defaults file for xterm, or even the Ubuntu Freetype library; I have built the Fedora xterm and my version of Freetype on Ubuntu and used the Fedora app-defaults, and the bad rendering is still there.

• I've directly set several fontconfig font rendering options that might be doing this without changing anything; at this point I haven't seen any difference with autohint, weight, embolden, or aspect (the last was a wild shot). Similarly, Xft X resources (cf) do nothing that I can see.

  (Forcing autohint=true actually makes the Fedora font rendering slightly but visibly darker while leaving the Ubuntu rendering unchanged for both the good and bad programs.)

• the problem doesn't happen with xterm on some FreeBSD machines I have handy; they render DejaVu Sans Mono the good way.

Clearly something mysterious is happening in the depths of the Ubuntu version of Xft or something it calls, but only if it's invoked in the right (or wrong) way. Unfortunately I don't think there's any good way for non-experts to see what font rendering choices are being made (the fontconfig library can be coaxed into some debugging output, but it's pretty much 'exports only' from what I can see), so I have no idea if I'll be able to figure out a solution that lets me use the font I want.

(Changing to gnome-terminal is not a solution for me.)

What you can find out about the memory usage of your Linux programs

Recently I wound up reading Linux Memory Reporting (via Hacker News), where Robert Haas talks about Linux's lack of clear reporting on process memory use. Today I'm going to sort of answer his question by covering what information Linux gives you about the various sorts of memory usage that you could be curious about. My primary focus is going to be on the numbers that you can get with ps, top, and smem. The background information on general Unix memory management will be helpful.

So, what you can get:

• the total amount of virtual address space that your process currently has allocated and mapped is the 'virtual size' of your process; ps reports this as VSZ and top reports it as VIRT. This includes anything the process has mapped, regardless of how it got there; the program's code, shared libraries, (System V) shared memory areas, mmap()'d files, mmap()'d private anonymous memory areas (which are often used by the C library for malloc()), everything.

  If your program is in a steady state but your VSZ keeps increasing, you have some sort of allocation leak. It may not strictly be a memory leak; you might be forgetting to unmap files or unload dynamically loaded code or something.

  (You can check at least some of this with lsof.)

• how much RAM would be immediately freed up if this process exited is smem's USS ('unique set size') field; this counts pages of RAM that the process is the only user of. These pages may be private pages (pages that will never be accessible by anyone else), or they may be shared pages that are only actively used by this process.

  (smem gets this information from the per-process smaps proc file.)

• how much RAM your program has looked at recently (which is roughly how much RAM it needs to be happy if it wasn't sharing anything) is the 'resident set size', reported as RSS by ps and smem and RES by top. The resident set size doesn't care whether or not some of that RAM is also used by other processes; each process counts it up separately.

  (In the terminology of my basic Unix memory management entry, a process's RSS is just how many page table entries in its virtual memory areas point to real RAM.)

  Your process's RSS increases every time it looks at a new piece of memory (and thereby establishes a page table entry for it). It decreases as the kernel removes PTEs that haven't been used sufficiently recently; how fast this happens depends on how much memory pressure the overall system is under. The more memory pressure, the more the kernel tries to steal pages from processes and decrease their RSS.

  If you have a memory leak it's routine for your RSS to stay constant while your VSZ grows. After all, you aren't looking at that leaked memory any more.

  A large RSS on an active system (one under memory pressure) means that your process touches a lot of memory (often rapidly) during its operation. A growing RSS means that it is increasing the amount of memory it touches. A constant RSS doesn't mean that the process is touching the same memory over and over; it just means that it's touching about the same amount of memory per unit time.

• the process's fair share of currently in use RAM is smem's PSS ('proportional set size') field. This prorates shared pages of RAM by charging each process for 1/Nth of the page, where N is how many processes currently have a page table entry for the page (the degenerate case is that you are charged the full page if you are the only user, ie this would be counted as part of your USS). Note that this is not how many processes have the shared resource mapped into their address space, it is how many processes have touched the page recently (ie, have it in their RSS). Mapping a shared resource is free (except to your VSZ); looking at it is what costs you here.

  It follows that the more processes actively look at pages of a shared resource, the lower each of their PSS goes for it (because more and more processes map the same pages from it).

  (Like USS, smem gets this information from the per-process smaps proc file.)

Because of how it's defined, summing the per-process PSS for a resource over all of the processes using that resource will tell you how much RAM that resource is using. Smem can do this (for some resources) with 'smem -m', although you need to know a certain amount about how Linux gives names to various resources in order to understand smem's output here.

(If you have all of the processes of interest running under a single userid, you can also use 'smem -u'. Smem doesn't currently have an option to aggregate reporting by program, so you can't do things like see how much memory your httpd processes are collectively using.)

As far as I know, Linux has no per-process or global number for how much of your virtual address size has ever been looked at (my second question in the six different meanings of memory usage). Nor can you get per-process information on how much memory the operating system might need to provide if your process wrote to everything it was entitled to (the sixth question), although you can get system-wide information on committed address space.

Top reports a SHR number but it's not clear to me how useful this is, partly because top doesn't document where it gets this information from. If I am reading the kernel code correctly, the most likely source is the (process) RSS for memory areas that were mmap()'d from files. I am not sure if this includes things like System V shared memory areas, and certainly it understates the potential sharing between, say, fork()'d processes. This is also only potential sharing, since it says nothing about whether or not any other process has mmap()'d the same object.

(Ie, if your single process mmap()'s a private two gigabyte file and then scans all of it, I believe that your SHR will be two gigabytes and change.)

Sidebar: answers to Bruce Momjian's questions

From his comment on Robert Haas's entry:

There are various methods for representing memory that is shared, either via SysV shared memory, fork's copy-on-write, or shared libraries. Does every process get charged the full amount, or do they split it among themselves, e.g. if five processes use shared memory, is each process charged 20% of the total size? (If another process attaches, does your percentage decrease?) What happens when you map in a large shared memory area but only access part of it? When do you stop using that memory?

Each process is charged the full amount to VSZ, but not to other numbers. When you map a large area but only refer to some of it, your VSZ goes up by the full amount but your RSS only goes up by the amount you access (and then goes down again at some rate if you don't access it and the system is under memory pressure). Your PSS is the only number that goes down if other people attach to the shared resource and also actually look at pages of that shared resource that you are also looking at (if they attach but don't look, your PSS doesn't change). If five processes all map the same shared memory segment but look at five different portions of it, each of them will be charged separately for their portion (their PSS for the segment will be the same as their USS); if they all look at the same portion, their PSS is 1/5th of the size of the portion.

(Your RSS never changes when people attach or detach from a shared resource.)

Notes on what Linux's /proc/<pid>/smaps fields mean

Because I was just digging around in the kernel source to determine this (it's a long story), here is some notes about what the fields of the smaps file mean and how they're calculated. The factory for this particular sausage is fs/proc/task_mmu.c (at least as of the current git tree).

For each VMA mapping that gets listed in smaps, the kernel walks all of the PTEs associated with it and looks at all of the known pages. Each PTE is then counted up:

• the full PTE size is counted as Rss.
• if the page has been used recently, it's added to Referenced.
• if the page is mapped in only one process it is labeled as private; its full size is added to Pss.
• if the page is mapped in more than one process it is shared and the amount it adds to Pss is divided by the number of processes that have it mapped.

(If the PTE is for something in swap it only adds to the Swap size.)

Note that a 'private' page is not quite as private as you might think. Because processes map pages independently of each other, it's possible to have a shared page that is currently mapped only by a single process (eg only one process may have called an obscure libc function recently); such pages are counted in 'private'.

The Size of a mapping is how much address space it covers.

If the mapping has been locked into memory via mlock() or the like, Locked is the same as Pss (ie, it is this process's fair share of the amount of locked memory for this mapping); otherwise it is 0 kB.

Given that looking at smaps requires walking the pages of all of the VMAs, I suspect that it's a reasonably costly operation. It'd probably be a bad idea to build a tool that did it a lot, especially if the tool scanned all processes in the system.

(Smem uses smaps, but it doesn't normally run repeatedly in the way that, say, top does.)

A Yum plugin I would like: using a local DVD as a repo source

It's become obvious to me that, to put it one way, Fedora 16 is where all of the update action is and Fedora 15 is not getting many changed packages (this is probably well known among people who actually pay attention to how Fedora is structured). This means that I really need to upgrade my home machine from Fedora 15 to Fedora 16. Because I'm sane I'll be doing this with a yum upgrade, which means that I need to get several gigabytes of all of those RPMs.

On my work machine, this is no particular problem because I have fast networking; I'm not going to notice fetching even gigabytes of data (and it goes basically as fast as the other end can feed it to me). At home, well, not entirely so much; I have a much better DSL downlink than I used to, but it is not really all that fast and I will definitely notice if it's in use.

What I would like to be able to do is use a local DVD as the source of as many of those packages as possible. The obvious DVD to use is the normal Fedora install DVD (hopefully most of the packages I need will be coming from the base Fedora 16 repository anyways instead of from the Fedora 16 updates). A plugin to do this would be useful for more than yum upgrades; among other things, you could also use it to easily add more packages after the upgrade (or after a from-scratch install from the DVD).

(This plugin would even be useful at work. Even though I do yum upgrades at work I often download the Fedora DVD image so that I can test in VMWare and do other things with it. It seems silly to download the same packages twice, once in a DVD image and once for a yum upgrade.)

While I think you can do this with a carefully created repos.d file, the plugin I'd really like would automatically notice things that look like mounted DVDs, check them for an install-DVD-like structure, figure out what Fedora version they're for, and create a repo on the fly as appropriate. This is probably a pipedream.

Using 'yum --downloadonly' overnight sidesteps a lot of the bandwidth issues for my specific case but it still feels like a wasted opportunity.

(I'd also like to be able to use a local DVD as a package source for mock's build environments. Repeatedly downloading base package sets every so often is not a really good use of my DSL link.)

PS: it's possible that what I want already exists and I just haven't found it. I think mock has some support for this, which I haven't investigated extensively as I don't often need to use mock at home.

How to use systemd to just run some stuff on boot

Suppose that you have some things that you want to get run when your system boots, much like rc.local. They aren't necessarily daemons, you don't want to wire them up to the whole systemd magic infrastructure, you just want to run something. Let us assume that you have some shell scripts for simplicity (if you don't, it's easy to convert what you need into one or more shell scripts).

When I was converting my old init.d stuff to systemd, here is how I did this:

[Unit]
Description=Run my stuff
After=network.target
Requires=network.target

[Service]
Type=oneshot
RemainAfterExit=True
ExecStart=/some/script
ExecStart=/some/other/script

[Install]
WantedBy=multi-user.target

(The systemd service that actually runs /etc/rc.d/rc.local is a bit different; see /lib/systemd/system/rc-local.service, at least on Fedora.)

If you want your rc.local equivalent to be started just as gettys and any graphical login manager are being started, it appears that you want to be after systemd-user-sessions.service. Most of the startup stuff I do doesn't need to be run that late but it does depend on networking being fully up so that the machine has an IP address and all its interfaces and so on.

(One of my wishlist items for systemd is the ability for services to depend on and be triggered on various sorts of network state changes. I suspect that the systemd people see this as more the job of NetworkManager and dbus in general, but NM is not something that I can use and I'd rather avoid having a second dependency and state change management system to handle dbus events when we already have a perfectly good general one in systemd.)

Nailing down RPM epoch numbers

As I mentioned here, RPM package version numbers have three components: an 'epoch' number, the package's upstream version, and an RPM package release version. The epoch number is there because upstream developers sometimes change version numbers in ways that break RPM's normal version comparison algorithm, or just plain use odd version formats.

(For example, I believe there's at least one program that increments its version by adding another digit of π to the end of the version number. This makes perfect sense in a way, but is hard for RPM version comparison to cope with.)

In that other entry I wrote that the default epoch number was '1' (and then a commentator corrected me to say that it was '0'). Both of us turn out to be wrong. The default RPM versioning is to have no epoch number at all. A RPM package that does not specify an epoch number does not have an epoch of '0'; it has no epoch at all, which is displayed by RPM as '(none)'. A package with no epoch number is a lower version overall than any epoch number, even 0.

On all of the RPM-based machines that I have sitting around, no epoch is by far the most common epoch. The next most common epoch is actually '1', not '0'; one theory I have about this is that a lot of people who build RPMs also think that the default epoch is 0, so when they specify an explicit epoch to override the upstream version number they start from 1.

Across all of the packages that I have handy on my machines, the highest epoch number is 50 (on aspell-en, on Fedora 14, Fedora 15, and Fedora 16). Other high epochs (on Fedora 16) are 32 (for bind's packages), 14 (tcpdump plus libpcap), and 12 (dhcp and aspell).

If you want to check your own RPM-based systems, appropriate rpm commands are:

rpm -qa --qf '%{E}\n' | sort | uniq -c | sort -nr
rpm -qa --qf '%{E}:%{N}\n'

The first will show you a count of epoch numbers; the second will show you the epochs for all of your packages (so you can find, for example, the package or packages at epoch 50).