Wandering Thoughts archives

The better way to install Sun's Java

I've been grumpy for quite a while about how Sun's Java RPM mutilates your system. Since I don't like my systems mangled, I've avoided installing Sun's Java on any of my systems unless I absolutely had to, and if I had to I tried to do it on an expendable test system. This has been vaguely annoying, because even in Fedora 10 there are things that don't work with the substitutes.

(One of them is the KVM over IP portion of the lights out management environment for Sunfire X2100s and X2200s. Since they have a very nice LOM and we've wound up with a bunch, this has been a bit annoying.)

When people offer their software packaged as RPMs, I reflexively pick that option. In the case of Sun, it turns out that this is a mistake, because the non-RPM version of Sun Java is much better. It is happy to be installed by a non-root user, in which case it doesn't mangle your system at all; instead it drops itself into a single directory hierarchy (and you can even move it around).

(It helps to know that the 'self-extracting file' seems to extract itself into your current directory.)

Right now I'm just interested in dealing with is 'JNLP' files (Java Webstart executables, which are run by javaws, which may be present only in the 32-bit version of Java), and for them I didn't even need to add things to my $PATH; I just pointed Firefox at the right executable once, and I was done.

All in all, I feel somewhat silly for not having looked at the non-RPM version a long time ago. If I had, I could have saved myself years of low-level irritation.

PS: according to here you may need some compat RPMs (I had them installed already).

Fixing your system after hitting the RAID growth gotcha

The easiest way to dig yourself out of the hole created by the RAID growth gotcha is probably to use a live/rescue CD. But let us suppose that you don't have one handy, which was the case for me yesterday. Further let us suppose that you have /boot as a separate filesystem, not as part of the root filesystem (if this is not the case, you absolutely need a rescue CD; sorry).

The basic goal is to rebuild a version of your current initrd that has an updated /etc/mdadm.conf that specifies that your root mirror has the right number of devices. Since we can't boot the system normally, we can't just bring it up, edit the real mdadm.conf, and regenerate the initrd with the normal tools; instead, you need to boot the system in a minimal mode and unpack, fix, and rebuild the initrd by hand.

First, you need some setup:

• you need to boot the system without the root mirror. Heed the cautions.

  Once booted, you'll probably want to mount /usr (read-only), if only so that you can read manpages.

• you need writeable scratch space to rebuild the initd; I mounted a tmpfs /tmp.

• you need /boot mounted read-write; if your /boot is mirrored, you'll have to assemble the software RAID first with appropriate mdadm invocations.

After that, it is relatively simple:

• make a scratch directory in /tmp (or wherever) and unpack your current initrd to it. Initrds are compressed cpio images, so this is something like:

  cd /tmp/t; zcat </boot/initrd | cpio -di

• edit the now-unpacked etc/mdadm.conf to have the right num-devices value for the software RAID with your root filesystem. You don't need to update the numbers for the other software RAID devices; they aren't started in the initrd.

• reassemble the initrd. On Fedora, the 100% authentic way to do this, exactly duplicating what mkinitrd does, is:

  echo nash-find . | /sbin/nash --force --quiet | cpio -H newc --quiet -o | gzip -9 >/tmp/initrd

• rename your current initrd to something else as a backup, and then copy your newly generated initrd into /boot with the right name.

At this point you can reboot and your system should come up as far as mounting the root filesystem. If you have other software RAID mirrors it will then throw a fit because none of them will successfully assemble either, and drop you into rescue mode. To fix this, edit /etc/mdadm.conf to specify the right num-devices number for all software RAID mirrors (including the root mirror). After you've done this, reboot and things should work.

(You may need to make the root filesystem read-write first with 'fsck /' followed by 'mount -o remount,rw /'.)

If you have no other software RAID mirrors, you still need to update /etc/mdadm.conf once the system has booted or you will get to go through this all over again after the next kernel update.

(You can see why I suspect a rescue CD would be easier. With a rescue CD you should be able to assemble and mount the root filesystem, chroot to it and set any other necessary filesystems up, edit /etc/mdadm.conf, and then run mkinitrd or the like.)

For safety, you probably want to rebuild your current initrd using the official tools after doing the full mdadm.conf update.

Booting a Linux system without a root mirror

Suppose that you have a system with a mirrored root filesystem and, for some reason, you need to boot it without the mirror being available. At one level, this is very easy; all you need to do is to find the partition of one of the mirrors and specify it as the filesystem root device on the kernel command line with 'root=/dev/sdXX'.

This works because while software RAID embeds its own metadata in the partitions, it puts the metadata at the end of the partition, not the start. So if you look at the partition without RAID, what you see is a normal filesystem (that happens to not be using all of the available space).

However, doing this comes with a huge warning. If you ever write to a filesystem accessed this way, you will have desynchronized the two sides of your mirror behind Linux's back; they will have different data when software RAID counts on them being exactly the same. What happens when the mirror is next reassembled ranges from unpredictable to explosive.

Therefor, you need to also boot your system in what I will call 'utterly single user mode', by supplying 'ro init=/bin/sh' as additional kernel arguments. It is not good enough to use 'single', as booting in normal single user mode writes things to the root filesystem. Also, if your root filesystem uses ext3 you should be very sure that your system shut down cleanly, because ext3 will write to the filesystem to replay the log even when you mount it read-only.

Booted this way, the system will then come up with /bin/sh running as the init process, PID 1, and the root filesystem mounted read-only. You can then mount other intact filesystems read-write, or just poke around. If you need scratch space, you can mount a tmpfs filesystem somewhere (such as /tmp).

If you actually need to write to the root filesystem, well, I think you get to break out a rescue CD of some sort and figure out how to assemble your mirror.

(There might be tricks you can play with mdadm to force the other side of the mirror to be seen as out of date, but I can't see anything obvious in the manpage. Maybe you could zero the RAID superblock on the other device and then re-add it later.)

A serious gotcha with growing software RAID devices

Suppose that you have a fully mirrored system and you want to expand your available space by migrating to a pair of larger hard drives. The obvious way to do this is fairly straightforward, assuming that you can connect all of your drives at once: partition the new drives (with larger partitions for at least some of your software RAID devices), add the new partitions to the appropriate MD device, let things resync, and then detach the old disks (and eventually physically remove them).

(Hopefully the obvious way is also the right way.)

One of the things that you will probably have to do in order to do this is to grow the number of devices in your mirrors. This is simple to do; 'mdadm -G -n 4 /dev/md0' will make it so that /dev/md0 can now have a four-way mirror (instead of the two-way one it's probably set up with).

If you do this, your system will probably fail to come up the next time you reboot it.

(Specifically, it will report that it is unable to set up the mirror for your root filesystem.)

In the old days, the Linux kernel automatically assembled software RAID devices itself. These days, this is done by your initrd running mdadm to assemble the root mirror, which uses information from an embedded copy of /etc/mdadm.conf. One of the pieces of information in mdadm.conf is how many devices each mirror has.

(You can see where this is going.)

mdadm will refuse to assemble a mirror, like say the one for your root filesystem, if the RAID superblocks on the disks say that the mirror has a different number of devices than the configuration file says it has, even if everything else matches. When you added more devices to your root mirror you probably didn't update /etc/mdadm.conf, and especially you probably didn't rebuild your initrd after doing that. And who does? The whole point of modern Linux initrds is that you don't have to rebuild them.

In my barely printable opinion, this behavior of mdadm is asinine, especially in combination with an uninformative initrd environment; it is robotically correct but practically useless. In an initrd environment, the only things mdadm should care about are whether the array UUID matches and the device can be assembled intact, and I am not entirely convinced about the UUID.

(Disclaimer: I know this happens on Fedora 8. It is possible that more recent versions of mdadm have fixed this, although I don't see anything in the changelog.)

How to set up your xorg.conf for RandR-based dual-headed systems

Theoretically, the usual way to set up your dual-headed system on a modern machine that uses the modern XRandR extension is to run xrandr once you've logged in to set things up. Suppose that you are not happy with this; you want your system to be properly dual-headed from the moment that X lights the display. This means that you need to have RandR set up in your xorg.conf.

(Alternately, you can hope that your Linux version has a graphical X configurator that understands how to set this up once and preserve it afterwards. Let us assume that you are not in this situation.)

Assuming that you already have an xorg.conf to start from, here is what you need, based on my experiences with an ATI Radeon.

(I am going to assume that you are familiar with xorg.conf syntax in general.)

• first, you need a "Monitor" section for each of your monitors. These days I believe that the only important thing in it is the Identifier, and whatever created your xorg.conf has probably already set these up for you anyways.

• next, you need to map XRandR outputs to monitors. This is done in the "Device" section for your graphics card with directives that look like:

  Option "Monitor-DVI-0" "monitor1"
Option "Monitor-VGA-0" "monitor2"

  What you put after the Monitor- is whatever xrandr prints out as the labels of your available outputs.

• finally, you need to tell XRandR how the outputs should be laid out. You do this in the "Monitor" section for at least one of your monitors, with the appropriate one of the LeftOf, RightOf, Above, or Below options:

  Option "LeftOf" "monitor1"

(Note that there are potential issues in some setups about how you map your physical monitors to card outputs.)

Now there's a potential complication. XRandR places outputs within a virtual screen, and this virtual screen has a maximum size that is generally decided by defaults inside your graphics driver. It is possible that the default maximum virtual screen size will not be large enough to lay out all of your displays. To see if this affects you, run xrandr and look at the information it tells you about your screen, for example:

Screen 0: minimum 320 x 200, current 1280 x 1024, maximum 1280 x 1200

At a maximum size of 1280x1200, this setup does not have a large enough virtual screen to be dual-headed with two standard 1280x1024 LCDs. To fix this you need to tell the driver to use a larger maximum virtual screen size, which is done with the Virtual directive in the "Screen" section's "Display" subsection:

Section "Screen"
  [...]
  Subsection "Display"
    Virtual 2560 1024
    [....]
  EndSubSection
EndSection

I currently specify the exact virtual screen size that I'm going to need. I don't know what the effects of specifying a larger than necessary virtual screen are (it probably depends on the driver), but since I'm hard-coding everything else anyways I figure that I might as well be conservative.

(If nothing else, I don't have the desktop space to add a third display.)

An irritation with Linux's 'mount -t nfs' output

The problem with 'mount -t nfs' is that it can't make up its mind about what NFS mount options it lists. Is its purpose to list all of the NFS mount options, or is its purpose to list only user-supplied ones? Right now the answer is some peculiar mixed blend; most of the time it lists only user-supplied options, but sometimes it will also list additional ones.

I care about this because I have programs that parse mount's output to see if we need to remount NFS filesystems to update their mount options. When mount -t nfs starts throwing in random additional options, the code explodes; it always thinks that the state is different and so always does remounts (which has various consequences).

I can deal with this in code, and have, but it's irritatingly inconsistent, especially because the list of sometimes-added mount options changes between different versions of mount. This is clearly going to make system upgrades somewhat more exciting in the future.

(And my code has to be somewhat complicated, because included in the extra options that mount can list are options that we sometimes set ourselves, so I can't just pretend that those mount options don't exist at all.)

Sidebar: what extra NFS mount options get added

The ones that I know of so far are:

• addr=..., for the IP address of the NFS server; always added.
• proto=udp, listed on Ubuntu 8.04 if the NFS server only supports NFS over UDP. (This doesn't happen on Ubuntu 6.06.)

Convenient ssh in Gnome, or 'my sshmenu wish comes true'

It turns out that I can get something that more or less does what I wanted sshmenu to do (per yesterday's entry). In email Dan Astoorian pointed me at the Gnome 'Command Line' applet and that it has macro capabilities; you can define a pattern and a command to run (using bits of the pattern) when the pattern matches the 'command' that you type. Thus, you can hijack it into a 'ssh session launcher' with suitable patterns.

The current patterns and commands that I've set up are:

(The RedText profile just has a foreground colour of #FF4500 (the classic X 'OrangeRed' colour value) instead of black, so that my root sessions stand out.)

This is actually better than just having a sshmenu text widget that I can type hostnames into, because I get significantly more flexibility. It helps that I already do something similar to this in my regular environment and actually like this peculiar form of mini-commands.

(If I was trying to make Gnome my normal environment I'd put all of the intelligence in a script and just feed all of these patterns to it, because it's much easier to make a script be super-smart than some regular expression matches, but that's too much setup work for what is really supposed to be quick customizations.)

I may well be doing this wrong (although it seems to work) or in a way that could be done better. In the usual Gnome style, the Command Line applet's macro stuff is underdocumented, with specific details about things like macro matching order and the like being notably lacking.