Wandering Thoughts archives

Some first impressions of Fedora Core 5

I've recently been playing with Fedora Core 5 (I know, I'm a bit behind the times) on a new Athlon 64 machine. In the spirit of my first irritations with Fedora Core 4, here are some very early, very preliminary impressions of Fedora Core 5:

• the hcid daemon (part of the Bluetooth stuff) consistently crashes on system shutdown on x86_64 machines. Since I don't have any Bluetooth stuff, I'll be removing the bluez-utils RPM, assuming the dependencies let me. (Bugzilla #186101 and #189464)

• gnome-terminal's cursor still blinks. Kconsole's does not. Advantage: KDE.
• gnome-terminal is no longer on the Gnome root menu. Kconsole is still on the KDE root menu. Advantage: KDE.

• it is surprisingly hard for even a relatively experienced person who's new to Fedora Core 5 to tell if an install is using KDE or Gnome just from the visual appearance. (Somehow I managed to de-select Gnome and select KDE in Anaconda, and then didn't notice for a while when I was using the system.)

• pirut, the graphical software manager, is pretty looking but pretty useless. I tried to install Gnome with it (once I noticed that I only had KDE), but it resolved dependencies with all the speed of a lazy snail and then produced very weird pukes once it got to the actual install phase; sometimes it claimed there were file conflicts, sometimes it claimed that something (that was already installed) couldn't be installed because a library was missing.

What I wound up doing was taking the list of RPMs pirut was going to install and feeding the list to 'yum install'. Reading the yum manpage (I should do this more often) suggests that I could have saved the work of the first step with

yum groupinstall "GNOME Desktop Environment"

(Possibly the name changed in FC5; 'yum grouplist' to see them. My FC5 machine is currently running memtest86+, so I can't check.)

• Anaconda's support for setting up RAID partitions is not so much primitive as almost completely backwards; it is primitive based ('make multiple RAID slices; make a RAID device from RAID slices; repeat endlessly') instead of task based ('make a partition of size X that is RAID-1 across these disks'). Some of its limitations are highly peculiar; for instance, it lets you clone one disk's partitioning to another but only if they have no non-RAID partitions.

This is the first time I've tried to use Anaconda to set up our standard mirrored system disks configuration. I'm not sure there will be a second time; the sheer boring repetition annoyed the heck out of me. (It's also strangely at odds with how task-oriented the basic partitioning is.)

(Unlike last time around, I haven't been playing with Anaconda upgrades or Kickstart installations, so I have no idea if they're better than with Fedora Core 4.)

Linux's %iowait statistic

The iostat manpage documents %iowait as:

Show the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.

It turns out that the manpage is wrong, which I found out by reading the kernel source because I was curious about what exactly it measured.

The actual definition of %iowait is the percentage of the time that the system was idle and at least one process was waiting for disk IO to finish. (This is true for both the 2.6 kernel and Red Hat's special 2.4 kernels with better disk IO statistics, including Red Hat Enterprise 3.)

(The actual kernel measure is the amount of time that each CPU has spent in each mode; it shows up in /proc/stat. iostat converts this to percentages.)

The difference may seem picky, but it's important because not all IO causes processes to wait. For example, Linux doesn't immediately flush data written to files to disk; it does it later, in the background, when it's convenient. Under the manpage's definition, this background flushing of data would take a system from %idle into %iowait, as would slowly paging out unused bits of programs.

This means %iowait is roughly the amount of time that your system could have been doing useful work if the disks were faster. A climbing %iowait is a danger sign that your system may be running into an IO bottleneck. A low iowait is not necessarily an indication that you don't have an IO problem; you also want to look at things like the number of processes shown as blocked ('b' state) in vmstat output.

(Finding potential disk IO bottlenecks and troubleshooting them is a really big field, so this is in no way comprehensive advice.)

Working on RPMs with quilt: an illustrated guide

I've mentioned quilt before; it's a package for managing a series of patches. One of the things it's really handy for is modifying source RPMs (which themselves are mostly a series of patches on top of a base source tree, hence the good fit). However, there's a bit of arcana involved in the process, so it's useful to have an example.

(Since my margin is only so big, I'm going to assume that you're broadly familiar with building RPMs and RPM specfiles. If not, look around rpm.org.)

First, you need to be using the sensible RPM build setup I described here. I'll also assume that you already have a source RPM that you want to modify; I'm starting with one for Liferea 1.0.9.

• cd sys/RPMS/SOURCES/liferea-1.0.9
• quilt setup *.spec
  This creates a liferea-1.0.9 source tree as a subdirectory and sets it up for quilt.
• cd liferea-1.0.9

• ln -s .. patches

You only need to create this symlink if the source RPM has no patches itself; otherwise quilt does it automatically (a future version of quilt may fix this oddity). The symlink makes it so that quilt's patch area is the same as the RPM source's patch area, which is exactly what we want.

We're now ready to start actually patching things:

• quilt new liferea-1.0.9-atom-title.patch
  Some style issues:
  • It's a good idea to start your patch series with the least controversial patches, the ones most likely to be accepted as bugfixes or whatever by the upstream developers, because the earlier they are the easier they're going to be to merge as standalone patches.
  • It's a convention to name RPM patches like this. The descriptive bit in the middle shouldn't be too long, but it should give someone skimming a list of patches some idea of what your patch is about.
• quilt add src/atom10.c
• edit src/atom10.c in your favorite editor
• quilt refresh: this actually generates the patch file (and will happily tell you about it).
• edit the patch file to add an explanation of what the patch is about and why it's necessary at the top; quilt will ignore (but preserve) this text. I tend to write the sort of explanation that I'd put in email to the upstream developers if I were to mail the patch off to them.

• cd .. to get back up to the RPM source directory
• edit the specfile to include the new patch. I usually start my own patches as Patch50 or so, so there's a big numbering gap between upstream source RPM patches and mine.
• rpmbuild -bb *.spec, and see if the patch actually works.

Now to illustrate the process of using quilt on an RPM with existing patches, I'll delete the liferea-1.0.9 directory and the series file and start again:

• quilt setup *.spec; cd liferea-1.0.9
  (We don't need to set up the patches symlink; now that the RPM already has a patch, quilt has done this for us.)
• quilt push -a

One of the few gotchas of using quilt with RPMs is that quilt defaults to starting off a newly set up patch area with no patches applied, instead of applying all of the patches from the RPM spec file and starting there. So if we were to just go ahead and start a new patch, it would wind up being at the front of the patch list, quite possibly causing some of the other patches to fail, instead of at the end. The quilt push -a applies all of the already existing RPM patches, so we're where we want to be.

(I've made this mistake, and it can be very irritating.)

• quilt new liferea-1.0.9-atom-times.patch
• quilt add src/atom10.c src/metadata.c
• edit the two files
• quilt refresh
• edit the new patch file to explain things, cd .., edit specfile, build, test, etc.

Interested people can find the actual patches on my liferea patches page.

PS: yes, this was sort of a cooked demo; I was less writing patches from scratch and more rolling forward existing patches I'd already written for an older version of liferea. Real quilt usage may have more flailing around with adding and removing files from patches and so on.

Sidebar: the basics of adding patches to RPM specfiles

Given a patch, like liferea-1.0.9-atom-title.patch:

1. go the initial section of the specfile and look for the Source: header (or SourceN: or a few other things), or any existing PatchNN: lines.
2. add a PatchNN: line for your patch, like 'Patch50: liferea-1.0.9-atom-title.patch'. Pick an NN value that is significantly larger than any existing ones, and remember it. (Also, to keep your sanity, keep the PatchNN headers ordered.)
3. skip down to the %prep section, which describes how to unpack the source and apply patches and what order this happens in. Add a %patchNN directive for your patch, after the %setup (and any existing %patch directives), like '%patch50 -p1 -b .atomtitle'. The %patch options are more or less the same as for patch(1).
4. skip down to the %changelog section; add your own entry.
5. skip up to the Release: header and change it; for example, I add .cks.N to most of mine. This clearly marks your RPM build as a non-standard version. (Usually my N is 1, but occasionally I have to respin an RPM and bump it up.)
6. Comment out any Packager: and Vendor: lines that already exist, because you're probably not them. Optionally add your own.

You're done.

Something to remember about networking restarts

I helped a coworker debug an interesting Linux problem today, and in the process wound up reinforcing something that I already more or less knew but hadn't really thought about for a while. Namely:

/etc/rc.d/init.d/network restart doesn't unload and reload the kernel modules for your network devices.

A networking restart will bring down and then restart all of your interfaces, reset parameters and so on, but the one thing it doesn't reset is kernel modules. They get loaded only once, the first time around.

Usually this doesn't matter. When it does matter is when you've changed some module parameters (or shuffled what kernel module maps to what ethN number); your changes won't take effect until the next time everything is unloaded and reloaded. This is especially pernicious for virtual network devices, like the bonding driver, where one of the module parameters is how many actual network devices to create.

The bonding driver is what we were scratching our heads over today. My coworker was trying to bring up a second bonding interface on a system that already had one configured, and it just wasn't going; no matter what he did, the bond1 device didn't appear. We added things to /etc/modprobe.conf, we restarted networking until the cows came home, we fiddled things, and nothing worked until we explicitly unloaded the module and reloaded it. We theorize that when things were initially set up, the module parameter that tells the bonding module to create more than one bonding device wasn't set.

(In the process I committed the classical mistake of doing 'network stop' while being logged in over the network. Fortunately this server had one of those 'remote console access' cards so we did not have to make a machine room visit.)

xiostat: better Linux disk IO statistics

Xiostat is the program I wrote to give us a faithful recounting of the Linux kernel's disk IO stats after we discovered the problems with iostat's numbers. I've now finally gotten around to putting it up on the web and making a page that explains how to run it and what its output means and so on.

The current version is a bit slapdash, but I have verified that it (still) works on bog standard 2.6 Linux kernels (and thus should work on Debian Sarge, Fedora Core 2+, etc) and Red Hat Enterprise Linux 4. It should also work on RHEL 3, but I don't have any handy RHEL 3 systems to test on right now.

Current information on xiostat's status will always be on the xiostat page, so check there for status updates from after this WanderingThoughts entry has been published.

(I am so habituated to using xiostat that when I wrote the original iostat entry I kept automatically typing 'xiostat' instead of 'iostat'.)

A bash irritation

Presented in illustrated form:

; cat foo
#!/bin/sh
cat "$@"
; ./foo /etc/termcap | sed 1q >/dev/null
./foo: line 2: 3632 Broken pipe  cat "$@"

This behavior is new in Bash 2, as far as I know. I find it extremely irritating, because programs having broken pipes is perfectly normal in Unix. Many filters and so on don't consume all of their input, and no other implementation of the Bourne shell reports this (as far as I know).

(I would not mind so much if this was only reported for interactive sessions; it is that bash spews out this message in shell scripts that irritates me so much.)

As the BASH FAQ covers in its section E2, this can be disabled when people compile bash. Debian apparently usually compiles bash this way (good for them); Red Hat does not (sigh). Of course this solution only helps people who can recompile and reinstall bash on all of the systems they want their shell scripts to run nicely on.

You can muzzle much of the verbosity of this by adding a do-nothing trap for SIGPIPE. Unfortunately you can't get it all; the best you can do is:

; cat foo2
#!/bin/sh
trap ':' 13; cat "$@"; exit 0
; ./foo2 /etc/termcap | sed 1q >/dev/null
Broken pipe

The message comes from bash itself. If you trap signal 13 to nothing at all, bash will set SIGPIPE to 'ignored' (SIG_IGN) for all of the processes in the pipe, which will cause them to see write errors instead of dying when the pipe breaks, which gets you:

; cat foo3
#!/bin/sh
trap '' 13; cat "$@"; exit 0
; ./foo3 /etc/termcap | sed 1q >/dev/null
cat: write error: Broken pipe

Which error message you prefer is a matter of taste. I tend to go for the foo3 case, because at least then I can remember why I am getting these strange messages (and grind my teeth about it yet again).

The other dynamic linking tax

I've already talked about one dynamic linking tax, but here's another one. Presented in illustrated form:

; cat true.c
#include <stdlib.h>
int main(int argc, char **argv)
{
    exit(0);
}
; cc -o true true.c
; cc -o true-s -static true.c
; diet cc -o true-d true.c
; ls -s true true-d true-s
  8 true    4 true-d  388 true-s
; strace ./true >[2=1] | wc -l
21
; strace ./true-s >[2=1] | wc -l
5
; strace ./true-d >[2=1] | wc -l
2

This is on a Fedora Core 2 machine. On a Fedora Core 4 machine the dynamically linked version makes 22 syscalls and the static linked glibc version makes nine.

strace's output always has the initial execve() that starts the program being traced and we're explicitly calling exit(), so the dietlibc version is doing the minimum number of system calls possible. Everyone else is adding overhead; in the case of dynamic linking, quite a lot.

This makes a difference in execution speed too. The dynamically linked glibc version runs 1.38 to 1.47 times slower than the dietlibc version, and the statically linked version 1.06 times slower. Admittedly this is sort of a micro-benchmark; most real programs do more work than this before exiting.

I ran into this while trying to measure the overhead of a program that I wanted to be as lightweight and fast as feasible. strace turned up rather alarming numbers for the overhead involved in glibc (although I believe it enlarges the overhead of system calls, so I'm not going to cite absolute numbers). So far I am being good and resisting the temptation to static link it with dietlibc.

Sidebar: just what's going on with glibc?

The statically linked glibc version also calls uname() and brk() (twice). The dynamically linked version, well, let's let a table show the story:

calls	syscall
5	old_mmap
3	open
2	mprotect
1	munmap
1	read
2	fstat64
1	uname
2	close
1	set_thread_area
1	brk
19	TOTAL

This table does not count the initial execve() or the final exit_group() (which glibc calls instead of exit()).

(Again, this is on a Fedora Core 2 machine. Your mileage will differ on different glibc versions. On FC4 the static linked glibc version does a uname, 4 brks, and a set_thread_area.)