The modern danger of locales when you combine sort and cron

February 24, 2019

I tweeted:

It never fails. Every time I use 'sort' in a shell script to be run from cron on a modern Linux machine, it blows up in my face because $LANG defaults to some locale that screws up traditional sort order. I need to start all scripts with:

LANG=C; export LANG

(I wound up elaborating on this, because the trap here is not obvious.)

On modern Linux machines, cron runs your cron jobs with the system's default locale set (as does systemd when running scripts as part of systemd units), and that locale is almost certainly one that screws up the traditional sort order (because almost all of them do), both for sort and for other things. If you've set your personal shell environment up appropriately, your scripts work in your environment and they even work when you su or sudo to root, but then you deploy them to cron and they fail. If you're lucky, they fail loudly.

This is especially clever and dangerous if you're adding a sort to a script that didn't previously need it, as I was today. The pre-sort version of your script worked even from cron; the new version is only a small change and it works when you run it by hand, but now it fails when cron runs it. In my case the failure was silent and I had to notice from side effects.

(Prometheus's Pushgateway very much dislikes the order that sort gives you in the en_US.UTF-8 locale. It turns out that my use of sort here was actually superstitious in my particular situation, although there are other closely related ones where I've needed it.)

I should probably make immediately setting either $LANG or $LC_COLLATE a standard part of every shell script that I write or modify. Even if it's not necessary today, it will be someday in the future, and it's pretty clear that I won't always remember to add it when it becomes necessary.

Written on 24 February 2019.
« Using default function arguments to avoid creating a class
Process states from /proc/[pid]/stat versus /proc/stat's running and blocked numbers »

Page tools: View Source, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun Feb 24 01:43:06 2019
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.