Wandering Thoughts archives

First irritations with Fedora Core 6

In the tradition of previous entries, some first irritations with Fedora Core 6. All of this is on an x86_64 upgrade from Fedora Core 5 on a machine with a stock configuration and stock GUI, and is mostly about the upgrade process.

• I had to manually specify noapic on my hardware. This is not too surprising, since the regular FC5 kernels now need it, but the FC5 install kernel was fine without it.
• at least if you have a non-RAID root device but software RAID on other devices, the upgrade process will bomb out, unable to mount them. Fortunately I did not have anything important on these partitions, so I was able to just comment them out of /etc/fstab. (On closer inspection, this is probably #188314; while my software RAID has two mirrors, I accidentally created it so that it thinks it should have three by fumble-fingering how I wanted to handle spare devices. The good news is that this won't affect my primary machine.)
• the dependency checking phase seems really slow. There is some indication (eg #211896) that this is fixable with 'rpm --rebuilddb' beforehand, but RPM database rebuilds usually make me nervous.
• upgrading is surprisingly slow, possibly as a side effect of the previous issue. Although I didn't time it, Anaconda claimed over 60 minutes and I can believe it.
• the update process spends too much time looking like it has hung, with the progress bar static and the mouse cursor a plain cursor instead of the spinning 'things in progress' one. Under the circumstances, this is a bit unnerving.

The heart-stopper is what happened when I commented my LVM filesystems back into /etc/fstab and rebooted: the system screeched to a dead halt with an inability to find any LVM volumes (and complaints about /dev/hdb, the DVD drive I'd installed from, not being there). After a great deal of hammering on things, I wound up deleting /etc/lvm/.cache to fix the situation. LVM appears to use this file to cache what devices to scan, and it got reasonably mangled at some point during the upgrade, probably when Anaconda was scanning things after it failed to bring up my RAID devices.

Overall I think I like the new DejaVu fonts (and I might not have really noticed the change if not for Pete Zaitcev's potential misgivings from some time ago). The new monospaced font is makes the difference between O and 0 much clearer, although I am not sure about 1 and l. (And poor ` stays pathetically hard to see, even harder than '.)

I prefer the FC5 default background; the new one is too dim and dark for my tastes. The FC5 one may have been relatively monochromatic, but at least it wasn't dark. The login screen is especially dark and makes me feel like I'm entering a cave (given the decorations, possibly one out of a video game).

The ATI X300 I have in the current test machine finally has hardware accelerated 3D and DRI (the website says that everything through the X850 based stuff should, but I haven't confirmed that in person yet). Compiz works fine, although on an upgrade one needs to remember to do 'yum install compiz' to have it available. Mind you, some of the default effects are more glitzy than good; I recommend turning off the optional settings in System → Preferences → Desktop Effects.

Sidebar: fixing the wrong number of RAID disks in a software RAID-1

If you have accidentally created a RAID-1 array with two real drives and one spare drive by doing:

mdadm -C /dev/md0 -l 1 -n 3 /dev/foo /dev/bar missing

instead of:

mdadm -C /dev/md0 -l 1 -n 2 -x 1 /dev/foo /dev/bar missing

(like I did), the solution is pretty simple:

mdadm --grow --raid-disks=2 /dev/md0

This can safely be done to a live software RAID device.

I have to dock Anaconda points for not telling me why it was not bringing up my RAID devices, especially if it is going to later bomb out because it can't mount partitions that are using them and is going to 'corrupt' my LVM configuration when I finally get things going. A warning would have saved me a lot of time and a certain amount of heart attacks.

A dump performance hint: the block size really matters

I have always more or less ignored dump's -b argument, assuming it was something only relevant for output to tapes (which I never do; even our backups that eventually wind up on tapes get sent to standard output for mangling by Amanda).

Except today I was getting disappointing dump speeds from a system with fast disks, so I decided to experiment with it. And boy have I been wrong.

(This is on a disk system that hits 69 MB/s on streaming reads.)

I got these numbers from a filesystem with a fair number of large files, but my home directory partition (complete with my MH folders) still did 40 Mb/s at 1024K. Since I am soon to dump and restore this machine to another one (don't ask) and I have a gigabit network link that I could use for this with some work, this is a quite timely discovery.

(Despite all the bad things that people say about it, the dump family remains my favorite tool for backing up and copying entire filesystems. I feel that dumping from outside the filesystem, with full access to the exact on-disk representation of the files you're working on, is the most reliable way in general to get an exact copy of everything.)

An irritating iptables limitation

I have a machine that runs both a caching DNS server (to answer local queries) and an authoritative DNS server; the caching server is bound to localhost:53, and the authoritative server to my-IP:53. Recently I wanted to let another machine use the caching DNS server.

The obvious general approach is to remap queries from the specific IP so that they hit the caching nameserver instead of the authoritative one. I can see two ways of doing this with Linux iptables: DNAT and REDIRECT. Unfortunately neither work:

• REDIRECT redirection doesn't work because it 'redirects the packet to the machine itself by changing the destination IP to the primary address of the incoming interface' (emphasis mine), not to localhost. In effect REDIRECT is a no-op here.
• DNAT redirection to localhost doesn't work at all, apparently because the kernel discards outside packets to localhost as 'martians', which matches the behavior I saw. Nor can you combine the DNAT rule with a SNAT rule to rewrite the source address to 127.0.0.1 to get around it, as far as I can tell.

Now, there are hacky ways around this. One of them would be to give the machine an additional completely private IP address, have the caching DNS server listen on it too, and use DNAT to reroute packets to that instead of localhost. But the need for that sort of workaround irritates me.

(The one useful thing to come out of this is that I had it pointed out to me that REDIRECT doesn't have to change the destination port; for some reason I had had the impression that it did.)

Why releases are important

Aigars Mahinovs recently wrote in passing in a blog entry:

So, from this perspective, there is nothing bad in money being paid to do this mundane and hard work, if we really, really need to release in a specific time frame (IMHO the only reason to release Debian in 2006, as opposed to 2008, is the Lars tattoo bet).

(Emphasis mine.)

This is the kind of thing that makes me reach for my editor and flail at the keyboard. There's a really good reason to release: it lets people use your distribution. (Or, in general, your software.)

Releases are important because they give a stable base for people to build their systems on top of. By stable I don't mean 'doesn't crash'; I mean 'doesn't change in a way that affects them'. This is especially important today, when many things that people want to run aren't just compiled and left alone but are built on top of other programs, ranging from Perl to Apache to database servers, that come from you. If the base of such a stack changes, everyone has an earthquake.

Updating an existing system isn't the only source of earthquakes; another is trying to build a second system to duplicate an existing one. There are ways around this, but they make you increasingly cut off from the actual distribution you are theoretically based on (and require you to do more and more work yourself).

(Or as the quote goes, 'you cannot step twice into the same river'. The more you need to, the more you have to build a dam.)

You might think that building duplicates of existing systems is an uncommon thing. But it's not; it happens every time someone says 'yeah, run <X>, my installation works great'. (Sure, they might get the same thing today. How about in three weeks, when they get around to trying <X> on their machine?)

(Old releases have their own problems, so I am not addressing that here.)

An Ubuntu astonishment

I just discovered that my Ubuntu test machine (running Dapper, not the current bleeding edge), installed with a base level of server packages, does not have cron installed.

It does, of course, have the /etc/cron.(daily weekly) directories, and cheerfully optimistic packages have populated them with various things that they no doubt expect to get run every so often in order to keep the system working smoothly.

(Evidently none of them actively depended on cron, or it would have been installed. Perhaps this is a bug. Fortunately exim4 did depend on cron and I actually read the list of packages that were going to be installed as dependencies, so cron is now explicitly listed as part of our additional core packages.)

Sidebar: a fun Ubuntu dependency chain

As an experiment, I started to install eximon4 (an X11 based Exim monitoring program). To my surprise, apt told me that it wanted to install dmidecode, which struck me as a rather absurd thing for an X11 program to require.

Except it doesn't, not directly. eximon4 requires libx11-6 (the main X libraries), which requires x11-common (the base X infrastructure), which requires laptop-detect (tries to detect if you're running on a laptop), which reasonably requires dmidecode.

(The dodgy hard dependency is x11-common to laptop-detect; I would rather not import laptop detection code on servers just so that I can run X clients. Current Debian packages have moved this dependency to the X server, which is more rational.)

How stable my AMD 64-bit systems have been

In a comment on a recent entry, Ryan asked me how stable my 64-bit Linux installs have been. The answer is kind of complicated, but the quick summary is that I am happy with the machines and they're stable.

Currently and in the recent past, I have dealt with three AMD 64-bit machines, two using ASUS M2N4-SLI motherboards, and one with a Tyan Tomcat K8E S2865ANRF, and all with 2GB of memory. The M2N4-SLI is a Socket AM2 motherboard, the Tyan is a Socket 939 board (and the motherboard that Sun used, in a modified form, in the first generation Ultra 20s).

The problems I've run into:

• The two M2N4-SLI machines have to use the noapic kernel parameter with current kernels. Without it, the machines lock up solid during boot (just before the kernel would normally print messages about 'PCI: Linking AER extended capability on ...'.)
• one M2N4-SLI has an ATI X800 GT PCIe graphics card. The original Fedora Core 5 drivers for it caused an instant reboot when they started; the current errata drivers are fine. (I don't have an X800 GT in any other machine, so it might be the card's fault, although the Tyan has an X800 GTO and it was fine.)
• the Tyan had a wierd problem where one bit in one word in the middle of memory would periodically wind up flipped in a particular memtest86+ test pass. We swapped the memory and nothing changed, so I opted not to try a motherboard swap and instead assumed that it was some SMM/SMI wierdness in the BIOS and used a Linux kernel feature to make sure that page would never be used.

In general I consider the ASUS problems about par for the course for the early days of a new chipset. And I'm not sure if any of these are exclusively 64-bit issues, since I haven't tried to see if any of them happen under a 32-bit install.

(One caution is that AM2 motherboards are apparently very picky about what memory they will take and how, especially if you want 800 Mhz memory, so sticking to the motherboard's recommended memory parts is highly recommended.)

Machines with 4GB or more of memory may run into various issues, which is one reason I stuck at 2GB.

Getting nice looking TrueType fonts on Fedora Core

Some TrueType fonts look significantly worse on some Linux distributions than on Microsoft Windows or MacOS. This is due to the usual reason that things suck on Linux: patents. TrueType fonts can have special hints for looking good at low resolution, but using those hints is apparently covered by some Apple patents in (some) countries with software patents.

(For the gory details, see here.)

Pretty much everyone on Linux uses the FreeType library to render TrueType fonts. FreeType has implementations of the patented stuff, but Fedora Core (and I believe some other Linux distributions) build the system FreeType libraries with it disabled due to the patent issues.

So to get nice looking TrueType fonts, all you need to do is rebuild the FreeType RPMs with the patented bytecode interpreter enabled, like so:

1. set up a sane RPM build environment.
2. get the FreeType source RPM and unpack it.
3. find the line near the start of freetype.spec that says '%define without_bytecode_interpreter 1.' Change the 1 to a 0.
4. change the Release field to mark your change; I usually add '.cks.0' or the like to it.
5. optionally, add an entry at the top of the %changelog section about this.
6. rpmbuild -bb freetype.spec
   (you may need to install some -devel RPMs that it needs)
7. update to your new FreeType RPM with 'rpm -U' or the like.
8. restart the X font server so that it's using the new FreeType shared library: quit X, then do /etc/rc.d/init.d/xfs restart, then restart X.

On an x86_64 machine, you probably want to rebuild both the native 64-bit RPM and the i386 version (and now you know why I was looking into cross-build RPMs yesterday). Rebuilding only the 64-bit RPM will leave any 32-bit programs that directly use FreeType with bad-looking TrueType fonts; this includes OpenOffice, ImageMagick, and a 32-bit Firefox (if you've installed one in order to get Flash et al working).

Cross building i386 RPMs on an x86_64 Fedora Core 5 machine

The more I do stuff on my x86_64 Fedora Core 5 machine, the more I come to understand that people must not have actually tried to use this stuff. The latest adventure was trying to build an i386 RPM.

The best tool for this is mock, a tool for rebuilding RPMs in distribution environments, like (in theory) 'Fedora Core 5 for i386'. In practice, it needs some clubbing. The main problem is that when mock sets up its build area, it winds up installing x86_64 packages instead of i386 packages, which means that compiling anything in the build area fails (it tries to build i386 binaries without necessary bits like crt1.o, or even shared libraries).

The chain of events seems to be that mock defers to yum, which defers to RPM stuff, which defaults to using whatever it finds in the host's /etc/rpm/platform if such a file exists. To get around this, you have to:

1. temporarily rename /etc/rpm/platform to something else, and then
2. run mock as 'linux32 mock ...', so that mock and everything it runs thinks it's in a 32-bit environment.

(I got this from a Fedora Extras mailing list post.)

Mach, the alternative to mock, does not suffer from this problem but has a number of its own that probably make it a worse choice for cross-building RPMs. (It's a better choice if you want a general compilation environment, but still not a great one.)

I can't really call this a mock bug, because it never actually claims to support cross-building. It just comes with tantalizing configuration files that make it look like it does.

Update, November 17th: this mock problem has been fixed in recent updates on at least Fedora Core 5 and FC6, making me happy and making cross-building much easier.

I ♥ Fedora Extras

The more I use Fedora Core 5, the more I've fallen in love with Fedora Extras. (I skipped straight from FC2 to FC5 and FCE came in with FC3, so this is the first time I've gotten to use it.)

This love is kind of silly, because Fedora Extras is not much more than Fedora catching up to Debian et al in having available a vast amount of prebuilt software. But boy is it convenient to have all that stuff around, ready to dip into to if I want to try out some bit of software; if it's reasonably popular, it's probably been packaged for Extras.

In a way the sheer size of Fedora Extras is daunting; there's so many packages that it's hard to just wander around and explore it. I wind up feeling more like I'm throwing darts at a phonebook (and usually I don't; I have some idea of what I want, and I use 'yum list foo' to pick out likely packages).

While I'm rambling, this sort of broad package availability is a great illustration of the benefits of reducing friction in computer stuff. While I'm perfectly capable of compiling stuff myself and even building RPMs, not having to do so has made me much more willing to just throw stuff in and poke at it, which has meant that I've played around with more things.

The other use of Extras is to supply the necessary packages that don't ship as part of Fedora Core any more; the package I use for dns queries in Python is part of Extras, as is mach. Somewhat to my surprise, so is the current version of xlock; I suppose I'm somewhat of a throwback in expecting that to be a core part of X. Someday I suspect that even xterm and xclock won't be parts of the core.

Sidebar: how to find out what RPMs you've installed from Extras

There isn't anything obvious in yum to do this, but all Fedora Extras RPMs have a vendor of 'Fedora Project', so:

rpm -q --qf '%{NAME} %{VENDOR}\n' | fgrep 'Fedora Project' | field 1 | sort

This says that I currently have 50 RPMs (out of 1551) from Fedora Extras. (It would be more if I was actively playing around with more things, but as it is I have relatively simple tastes.)

Sidebar: why not xscreensaver?

It's not like Fedora Core 5 doesn't have a screen lock program; it ships with xscreensaver. So why don't I use that instead?

First, history. xlock is just what my window manager configuration uses, because I started doing it before xscreensaver existed. I'm stubborn enough to stick with it, even when it means I have to install an extra package.

Second, I don't want fancy graphics screensavers; I just want a black screensaver that uses zero CPU, a habit I started back when I was taking part in the RSA-129 factoring challenge. I'm sure xscreensaver has a module that does this; I just haven't felt like going through the effort of finding it.

Some numbers for modern disk performance

Since I currently have a semi-test system floating around, I decided to collect some numbers on what sort of disk performance I can expect from a modern SATA-based system. The quick summary is that it's pretty good.

I'm only really interested in three numbers: streaming writes, streaming reads, and random IOs per second. Fortunately bonnie++ will give me those, along with a bunch of other numbers that I don't really care about.

My test system has two Seagate 7200.10 320GB SATA drives, identically partitioned, and all of the testing used a pair of identical (and otherwise empty) 58 Gb partitions.

(Disclaimer: I won't claim that these are scientific benchmarks, carefully statistically analyzed to make sure that they're valid; I don't have that much time and patience.)

All LVM partitions were using 32 Mb extents, which is the default on Fedora Core 5.

Some interesting things show up in these numbers: for example, note that using LVM apparently somehow improves your seek performance (assuming that bonnie is measuring this correctly). In general, LVM seems to at least not hurt compared to the alternative, and may even work better, which is reassuring since I'm setting up this system with LVM, having given in to the temptation.

Sidebar: the test system

The test system is an ASUS M2N4-SLI (running with noapic) with an Athlon 64 X2 4600+, 2G RAM, and the aforementioned Seagate SATA drives, using the x86_64 Fedora Core 5 with the current set of updates. bonnie++ was run with just '-r 2048', so it used 4GB per instance. The system was running a CPU cruncher per core, but was otherwise idle.