Wandering Thoughts archives

A problem with gnome-terminal in Fedora 21, and tracking it down

Today I discovered that Fedora 21 subtly broke some part of my environment to the extent that gnome-terminal refuses to start. More than that, it refuses to start with a completely obscure error message:

; gnome-terminal
Error constructing proxy for org.gnome.Terminal:/org/gnome/Terminal/Factory0: Error calling StartServiceByName for org.gnome.Terminal: GDBus.Error:org.freedesktop.DBus.Error.Spawn.ChildExited: Process org.gnome.Terminal exited with status 8

If you're here searching for the cause of this error message, let me translate it: what it really means is that your session's dbus-daemon could not start /usr/libexec/gnome-terminal-server when gnome-terminal asked it to. In many cases, it may be because your system's environment has not initialized $LC_CTYPE or $LANG to some UTF-8 locale at the time that your session was being set up (even if one of these environment variables gets set later, by the time you're running gnome-terminal). In the modern world, increasing amount of Gnome bits absolutely insist on being in a UTF-8 locale and fail hard if they aren't.

Some of you may be going 'what?' here. What you suspect is correct; the modern Gnome 3 'gnome-terminal' program is basically a cover script rather than an actual terminal emulator. Instead of opening up a terminal window itself, it exists to talk over DBus to a master gnome-terminal-server process (which will theoretically get started on demand). It is the g-t-s process that is the actual terminal emulator, creates the windows, starts the shells, and all. And yes, one process handles all of your gnome-terminal windows; if that process ever hits a bug (perhaps because of something happening in one window) and dies, all of them die. Let's hope g-t-s doesn't have any serious bugs.

To find the cause of this issue, well, if I'm being honest a bunch of this was found with an Internet search of the error message. This didn't turn up my exact problem but it did turn up people reporting locale problems and also a mention of gnome-terminal-server, which I hadn't known about before. For actual testing and verification I did several things:

• first I used strace on gnome-terminal itself, which told me nothing useful.

• I discovered that starting gnome-terminal-server by hand before running gnome-terminal made everything work.

• I used dbus-monitor --session to watch DBus messages when I tried to start gnome-terminal. This didn't really tell me anything that I couldn't have seen from the error message, but it did verify that there was really a DBus message being sent.

• I found the dbus-daemon process that was handling my session DBus and used 'strace -f -p ...' on it while I ran gnome-terminal. This eventually wound up with it starting gnome-terminal-server and g-t-s exiting after writing a message to standard error. Unfortunately the default strace settings truncated the message, so I reran strace while adding '-e write=2' to completely dump all messages to standard error. This got me the helpful error message from g-t-s:

  Non UTF-8 locale (ANSI_X3.4-1968) is not supported!

  (If you're wondering if dbus-daemon sends standard error from either itself or processes that it starts to somewhere useful, ha ha no, sorry, we're all out of luck. As far as I can tell it specifically sends standard error to /dev/null.)

• I dumped the environment of the dbus-daemon process with 'tr '\0' '\n' </proc/<PID>/environ | less' and inspected what environment variables it had set. This showed that it had been started without my usual $LC_CTYPE setting (cf).

With this in hand I could manually reproduce the problem by trying to start gnome-terminal-server with $LC_CTYPE unset, and then I could fix up my X startup scripts to set $LC_CTYPE before they ran dbus-launch.

(This entry is already long enough so I am going to skip my usual rant about Gnome and especially Gnome 3 making problems like this very difficult for even experienced system administrators to debug because there are now so many opaque moving parts to even running Gnome programs standalone, much less in a full Gnome environment. How is anyone normal supposed to debug this when gnome-terminal can't even be bothered to give you a useful error summary in addition to the detailed error report from DBus?)