Wandering Thoughts archives

An interesting filesystem corruption problem

Today we had a fun problem created by a combination of entirely rational find optimizations and a corrupted, damaged filesystem.

An important Linux server took some kind of hit that turned some files into directories (with contents, presumably stolen from some other poor directory). We found some but were pretty sure there were others lurking out there too, and wanted to do our best to find them. (If only to figure out what we needed to restore from the last good backups.)

As it happens, most of the actual files on this filesystem have some sort of extension, and pretty much all directories don't. So, I made the obvious attempt:

find /hier -name '*.*' -type d -print

Much to my surprise, this didn't report anything, not even the files we already knew about in /hier/foo/bar.

Okay, first guess: I happened to know that find optimizes directory traversals based on its knowledge of directory link counts, so if the count is off find will miss seeing directories. A quick check showed that /hier/foo/bar had the wrong link count (it only had two links, despite now having subdirectories). Usefully, find has a '-noleaf' option to turn this off (it's usually used to deal with non-Unix filesystems that don't necessarily follow this directory link count convention).

But that didn't work either. Fortunately I happened to know about the other optimization modern Unixes do for find: they have a field in directory entries called 'd_type', which has the type of the file (although not its permissions). If files had gotten corrupted into directories, it would make sense that the d_type information in their directory entries would still show their old type and make find skip them.

A quick d_type dumper program showed that this was indeed the case. This also gave us a good way to hunt these files down: walk the filesystem, looking for entries with a mismatch between d_type and what stat(2) returned.

In retrospect, I have to thank find for biting us with these optimizations; it led me to a better way to find the problem spots than I otherwise would have had.

(And writing a brute force file tree walker, even in C, turns out to be not as much work as I thought it would be.)

This is of course a great example of leaky abstractions and how knowing the low-level details can really matter. If I hadn't been well read enough about Unix geek stuff, I wouldn't have known about either find optimization and things would have been a lot more hairy. (I might have found -noleaf with sufficient study of the manpage, but that wouldn't have been enough.)

Why I am irritated with evince

Evince is in general a nice PDF viewer (technically it supports other formats too), much nicer and far less irritating than Adobe Acrobat. (For a start, evince actually quits when I click on the 'close window' button; Adobe Acrobat presumes that I wanted it to clear the current document.)

The problems come in when I tried to do something besides view a document. (Perhaps evince was jealous that I wasn't satisfied with it.)

First, evince silently aborts printing if I have it print a document and then tell it to quit once it seems to have finished. There is no sign that it is still working on the print request and no warnings. This is teeth grindingly irritating, especially because when a print job goes missing my immediate assumption is that something in the print system ate it.

(There are lots of grues in print systems, especially around here.)

Attention interface designers: if the user can't tell when it's safe to quit and when it's not, you've screwed up.

Once I worked this out, I was lucky enough to do a test print run of only a small section of the 500+ page IBM Tivoli Redbook I was trying to print. It was lucky because it turns out that evince defaults to forced non-duplex printing and thus I only wasted 20 pages instead of 500 odd. The sheer twisted genius of this choice is breathtaking; I can think of few defaults that so carefully calculated to surprise and irritate people and to waste reams and reams of paper.

(What evince should do is to not override the printer's default settings for duplexing and the like. I know it does, because I was printing to a destination that defaults to duplexing.)

I'm not even sure I can set evince to default to (forced) duplex printing; I may just have to remember to set the ticky-box in a non-default tab of the print dialog every time I want to print something.

Why apt-get is not my favorite application (part 2)

In another nutshell:

# apt-get install fvwm
...
Suggested packages:
  fvwm-themes m4 menu wm-icons imlib-progs
Recommended packages:
  fvwm-icons imlib11
...

So I abort the apt-get when offered the chance, because I want the useful bits too.

# apt-get install fvwm fvwm-icons imlib11 fvwm-themes m4 menu wm-icons imlib-progs
[...]
Package fvwm-themes is not available, but is referred to by another package. This may mean that the package is missing, has been obsoleted, or is only available from another source
E: Package fvwm-themes has no installation candidate

# apt-get install fvwm fvwm-icons imlib11 m4 menu wm-icons imlib-progs
[...]
Package imlib-progs is not available, but is referred to by another package. This may mean that the package is missing, has been obsoleted, or is only available from another source
However the following packages replace it:
  imlib11-dev imlib-base
E: Package imlib-progs has no installation candidate

Argh!

Dear apt-get and the Ubuntu/Debian packaging system: please stop suggesting and recommending things that I can't actually install. And if you must have these problems, please report them all at once, not in a drip by drip, one by one Chinese water torture.

While I am griping, it would be nice if apt-get had a command-line option for 'install the recommended and/or suggested packages too'. (You really need two options, one for recommended, one for suggested.)

(All of these packages come from the official Ubuntu package system.)

The quick secret to bootable USB keys

We got asked this in email recently, so I am going to throw my answer up here: the quick secret of making bootable USB keys is to know that USB keys are hard drives.

The person who asked the question described what they'd done as:

In short, I've copied the files from the isolinux directory on the installation CD to the USB drive, renamed isolinux.cfg to syslinux.cfg, erased isolinux.bin, unmounted the USB drive, and run syslinux /dev/sda1.

The one magic ingredient missing is that the USB key itself needs a boot block, just like a real hard drive does. (It does not need to be a very sophisticated boot block and boot loader, since its usual purpose is to immediately pass control to the 'real' boot stuff on a partition.)

How we got the boot block in our setup is that I just installed GRUB on the 'hard drive', configuring it to immediately start the syslinux in the single partition. There are probably easier ways, but I don't know them and I was in a hurry at the time and GRUB was handy.

The other way to deal with this, and sometimes the more convenient way, is to not use partitions on the USB key at all and simply dump stuff directly on the entire disk, treating it as a giant floppy (using /dev/sda instead of /dev/sda1). For Fedora Core, you can just dd the diskboot.img file (found in the images directory on the CDs or DVD) to the USB key's SCSI device straight.

(Disclaimer: I've only tried this with exactly one USB key on one test machine that I had handy, so I can't say that this is extensively tested, and I have a vague memory that I wound up using the partitioning approach because at some point I had problems with the non-partitioned one.)

Distributions: keep your hands off vi

Dear Linux distributions: if I want to use a freaky super-intelligent editor I am perfectly able to fire up emacs myself, or even type 'vim' instead of 'vi' (honest, the extra letter is not too much for me to type). I want vi. You know, the simple and predictable thing that system administrators like because it behaves.

As an aid to your planning, here is a minimum feature for any editor called vi:

When I paste things into an alleged vi in insert mode in a terminal window in X, you put them in exactly as they are. No more, no less.

If your 'vi' does anything else, you are not being helpful, you are causing me to violently throw your distribution at the wall because you have just screwed up what I was trying to do.

(I would prefer that you not mangle my typing either, but not mangling my copy and pastes in X is a minimum standard.)

(I do not mind whatever distributions choose to do with things called 'vim' and the like. They can be as freaky super-intelligent as they want. My objections are strictly for what happens when I innocently type 'vi'.)