Wandering Thoughts archives

Bash is letting locales destroy shell scripting (at least on Linux)

Here, let me present you something in illustrated form, on a system where /bin/sh is Bash:

$ cat Demo
#!/bin/sh
for i in "$@"; do
  case "$i" in
    *[A-Z]*) echo "$i has upper case";;
  esac
done
$ env - LANG=en_US.UTF-8 ./Demo a b C y z
b has upper case
C has upper case
y has upper case
z has upper case
$ env - LANG=en_US.UTF-8 /bin/dash ./Demo a b C y z
C has upper case
$ env - ./Demo a b C y z # no locale
C has upper case
$

I challenge you to make sense of either part of Bash's behavior in the en_US.UTF-8 locale.

(Contrary to my initial tweet, this behavior has apparently been in Bash for some time. It's also somewhat system dependent; Bash 4.2.25 on Ubuntu 12.04 behaves this way but 4.2.45 on FreeBSD doesn't.)

There is no two ways to describe this behavior: this is braindamaged. It is at best robot logic on Bash's part to allow [A-Z] to match lower case characters. It is also terribly destructive to bash's utility for shell scripting. If I cannot even count on glob operations that are not even in a file context operating sanely, why am I using bash to write shell scripts at all? On many systems, this means eschewing '#!/bin/sh' entirely because (as we're seeing here) /bin/sh can be Bash and Bash will behave this way even when invoked as sh.

(I have to assume that not matching a as upper case is a Bash bug but that the rest of the behavior is intended. It makes more sense than the other way around.)

What Bash has done here is to strew land mines in the way of my scripts working right in what is now a common environment. If I want to continue using shell scripts I have to start trying to defensively defeat Bash. What will do it? Today, probably setting LC_COLLATE=C or better yet LC_ALL=C. In all of my scripts. I might as well switch to Python or Perl even for small things; they are clearly less likely to cause me heartburn in the future by going crazy.

There's another problem with this behavior, which is that it is not what any other POSIX-compatible shell I could find does (on Ubuntu 14.04). Dash (the normal /bin/sh on many Linuxes), mksh, ksh, and even zsh don't match here. This means that having Bash as /bin/sh creates a serious behavior difference, not just adds non-POSIX features that you may accidentally (or deliberately) use in '#!/bin/sh' scripts.

(Yes, yes, I've written about this before. But the examples back then were vaguely sensible things for locales to apply to. What is happening in the Demo script is very, very far over the line. What is next, GNU grep deciding that your '[A-Z]' should match case-independently in some locales? That's just as justified as what Bash is doing here.)

PS: This is actually making me rethink the idea of having /bin/sh be Bash on our Ubuntu machines, which is the case for historical reasons. The pain of rooting out bashism from our scripts may be less than the pain of dealing with Bash braindamage.

Sidebar: the bug continues

If you change the [A-Z] to [a-z] and try Demo with all upper case letters, it will match A-Y but think Z doesn't match. This is symmetrical in what you could consider a weird way. A quick test suggests that all other letters besides 'a' (in the [A-Z] case) and 'Z' (in the [a-z] case) match 'correctly', if we assume that a case independent match is correct in the first place.

Because I was masochistic tonight this has been filed as GNU Bash bug 108609 (tested against bash git tip), although savannah.gnu.org may have eaten the actual text I put in (it sent the text to me in email but I can't read the text through the web). My bug is primarily to report the missing 'a' and 'Z' and only lightly touches on the whole craziness of [A-Z] matching any lower case characters at all, so I encourage other people to file their own bugs about that. I have opted for a low-stress approach myself since I don't expect my bug report to go anywhere.

Why Solaris's SMF is not a good init system

An init system has two jobs: running and supervising services, and managing what it runs and supervises. SMF is a perfectly good init system as far as the former goes, and better than some (eg the traditional System V system). It is the second job where SMF falls down terribly because it's decidedly complex, opaque, fragile, and hard to manipulate. The result is a very divided experience where as long as you don't have to do anything to SMF it's a fine init system but the moment you do everything becomes immensely frustrating.

Here is an illustration of how complex, opaque, and fragile SMF is. The following is a script of commands that must be fed to svccfg as one block of actions in order to make two changes to our OmniOS systems: to start syseventd only after filesystem/local (it normally starts earlier), and to start ssh after filesystem/minimal (ie very early in the boot process, so if things go wrong we have system access).

select svc:/system/filesystem/local
delpg npiv-filesystem
select svc:/milestone/devices
delpg devices
select svc:/milestone/single-user
delpg syseventd_single-user
select svc:/system/sysevent:default
addpg filesystems-dep dependency
setprop filesystems-dep/grouping = astring: "require_all"
setprop filesystems-dep/restart_on = astring: "none"
setprop filesystems-dep/type = astring: "service"
setprop filesystems-dep/entities = fmri: "svc:/system/filesystem/local:default"
select svc:/network/ssh
setprop fs-local/entities = fmri: "svc:/system/filesystem/minimal"
delpg fs-autofs
end

There are two obvious things about this sequence of commands, namely that there are quite a lot of them and they are probably pretty opaque if you're not familiar with SMF. But there are several other things that are worthy of mention. The first is that it is actually fairly difficult to discover and work out what these commands should be and need to be; I had multiple false steps and missteps during the process. Many of the names involved in this process are arbitrary, ie up to the individual services to decide on and as you can see many of the services have chosen different names. These names are of course not documented and thus presumably not officially supported.

(Nor do the OmniOS SMF manpages discuss how your changes interact with, say, applying a package update for the packages you've manipulated.)

The next inobvious thing is that if you get some of these wrong, SMF will thoroughly blow up your system by, for example, detecting a dependency cycle and then refusing to have any part of it instead of trying some sort of fallback cycle-breaking in order to allow your system to boot to some extent. Nor does SMF prevent you from creating a dependency cycle by (for example) refusing to commit a service change that would set up such a cycle; instead it just tells you that you've made one somehow. This is why I call managing SMF a fragile thing.

Oh, and the third inobvious thing is that there are several ways to do what I've done above, all of them probably roughly equivalent. At least one thing I've done is a convenient hack instead of what would be the theoretically 'correct' way; I've done the hack because the theoretically correct way is too much of a pain in the rear. That by itself is a glaring problem indicator, as doing the correct thing should be the easiest approach.

(The hack is that instead of deleting ssh's property group for its dependency on filesystems/local and creating a new property group for a new dependency on filesystem/minimal, I have instead rewritten the specific service that 'fs-local' depends on and thus its name is now kind of a lie. But this change is one line instead of six lines, making it an attractive hack.)

To be manageable, an init system needs to be clear, well documented, and easy to use. You should be able to easily discover what properties a service has, what properties it can have, how these affect its operation, and so on. It should be obvious or at least well documented how to change service start order (because this is a not uncommon need). For dependency-based init systems without a strict ordering, it should be easy to discover what depends on what (including transitively) and either impossible or as harmless as possible to create dependency cycles. It should not require a major obscure bureaucracy to change things, nor hunts through the Internet and Stack Overflow to work out how to do things.

SMF is not a success at any of these, especially being easy to use (about the only thing that is simple in SMF is simply disabling and enabling services). That is why I say that it is not a good init system. If I had to describe it in a nutshell, I would say that SMF is a perfect illustration of what Fred Brooks calls the second system effect. People at Sun clearly wanted to make a better init system that fixed all of the problems people had ever had with System V init, but what they put together is utterly over-engineered and complex and opaque.

(I also have to mention that SMF falls down badly on the small matter of managing serial port logins. Doing this in SMF is so complicated that no one I've talked to can tell me how to successfully enable logins on a particular serial port. Really. This is yet another sign that something is terribly wrong in how SMF is configured and manipulated, even if it's perfectly fine at starting and restarting services once you can configure them.)