Wandering Thoughts archives

Forcing sort ordering in Unix shell scripts

There are a number of situations in Unix shell scripts where you know the order that the lines in your output should be in, but you can't produce it in that order and nothing in your normal output is useful to sort on. One common case is awk scripts that accumulate information into arrays and then dump it out in their END blocks with 'for (key in array) ...'; array keys are, of course, produced in no particular order.

In this situation, some people go to bat against sort with complex key specifications and the like. I'm lazy, so my usual solution is simple: I add an extra field at the start of the line that has a useful key for sort, sort the output, and then run the output through a second awk script to remove the first field (and often to do the final neat formatting of the output).

A variant of this trick can be used to re-order output lines that are easiest to produce together. A snippet from one of my shell script is perhaps the best illustration:

.... |
(while read fs typ opts; do
     echo 0 unshare $fs
     echo 2 share -F $typ -o $opts $fs
 done;
 echo 1 sleep 60) |
sort -n | sed 's/^[012] //' | ...

What this script wants to do is unshare a bunch of filesystems, wait some time, and then reshare them all. However, it gets all of the necessary information about each filesystem once, in a big chunk, so it is most natural to generate both the unshare and the reshare command at the same time. To insert a sleep in the middle we add a hidden field and just pick entirely artificial keys for all the lines such that sort will put all of the unshares first, the sleep second, and then all of the shares third.

(Disclaimer: this is clearly at least related to Perl's Schwartzian transforms, although I think that they're not quite the same thing, and I probably picked up this shell idiom from somewhere.)