Wandering Thoughts archives

Why Unix filesystems precreate lost+found directories

Here's a sensible question about the lost+found directories that various Unix filesystems stick in the top directory: why do they have to exist all the time? Yes, clearly fsck needs somewhere to put recovered files, but in theory it could just create a lost+found directory when needed, and otherwise you wouldn't have it cluttering up the filesystem.

Part of the answer is that lost+found isn't just a normal empty directory; instead, it's usually substantially bigger, despite having nothing in it.

The full answer is simple: because having a pre-existing lost+found directory means changing as little of the filesystem as possible during an fsck recovery. With an existing lost+found directory that has a bunch of empty directory entries, all fsck has to do to recover a file is fill in some data and rewrite a block.

Without an existing lost+found directory with empty entries, fsck would have to do a lot more; it would have to allocate an inode for the directory, allocate at least one data block for it, and add a directory entry to the filesystem's root directory (which might require allocating another data block and extending the directory). Of course, these changes would require checking and updating various potentially damaged filesystem data structures.

You could make an fsck that would do all this but it would clearly be more complicated, and complicated operations are undesirable in a filesystem recovery tool. Setting everything up ahead of time is much simpler.

(This entry was inspired by this comment.)

The rules for combining things with Bourne shell 'here documents'

Shell here documents are relatively simple to use on their own, but it's usually head-scratching to combine them with other things (redirections, pipelines, or conditionals). To save me having to do the experimentation and manpage reading the next time, here's the simple version of the rules: here documents always start at the next line.

So you can write any number of other things on the same line as the command with a here document:

cat <<EOF | ssh host ...
foo
EOF

But if you need to continue the whole thing on additional lines, you must write the following commands after the here document:

ssh host ... <<EOF ||
foo
EOF
	echo "ssh failed"

However, 'next line' here means the next line as seen by the Bourne shell after backslash-escaped newlines, so you cannot write the last example with the ssh line ended with a '\' and the '||' at the start of the echo line. (This is a pity, as I think it would be more readable. But frankly, combining here documents and multiple commands this way is never going to be terribly readable.)

Because of this, other redirections can go anywhere on the command line with the here document, as can arguments and so on; 'ssh <<EOF host ... >logfile 2>&1' is perfectly legal. The maximally perverse version of this is:

<<EOF ssh host ... 
foo
EOF

My opinion is 'don't do that'; put the here document redirection last on the line if at all possible, to avoid confusing other people.

Somewhat to my surprise it's possible to specify multiple here documents on the same line. If you do this crazy thing, the shell expects them to be in left to right order, eg:

sed 's/^/a: /' <<EOF && cat <<EF2
foo
EOF
bar
EF2

But, really, don't do this.