Wandering Thoughts archives

How to accidentally reboot a server

This is a little war story.

To start out, determine that you need to add more memory to one of your login servers. This requires taking it down, which requires getting all of its users to log off first. Then go through the following steps:

1. Keep new logins off the system by changing your automated password management system to give all non-staff accounts a login shell that just prints out a 'this system is under maintenance, please use another one' and then quits.

   (Otherwise people will keep logging in to the server and you'll never, ever get a moment where there is no one on. Especially during working hours.)

2. Mail all of the current users to ask them to log off.

3. Get email from one user saying 'you can kill all of my processes on the machine'.

4. Do this in the obvious way. From an existing root session:

   /bin/su user
kill -1 -1

5. Have all of your (staff) sessions to the machine suddenly disconnect. Get a sinking feeling.

Perhaps you can spot the mistake already. The mistake is that the su to the user did not actually work. The user had a locked login shell, so all the su did was print a message and then dump me back into the root shell. Then I ran the 'kill -1 -1' as root and of course it SIGHUP'd all processes on the machine, effectively rebooting it.

(It didn't actually reboot and in fact enough stayed up that I could ssh back in, which surprised me a little bit.)

I should have used '/bin/su user -c "kill -1 -1"' or in fact one of the ways we keep around to do 'run command as user no matter what their shell is'. But I didn't take the time to do either of them.

(On the 'good' side, we got to immediately add more memory to that server.)

My Cinnamon desktop customizations

As I mentioned in the previous entry on my laptop environment, I'm now using a lightly customized Cinnamon desktop on Fedora 19. The major customizations have been to get back something equivalent to sshmenu and the Gnome 2 mini-commander applet. The results have been quite good; I feel as productive with my Cinnamon setup as I did with Gnome 2 on Fedora 14 (and many more things work on Fedora 19).

My mini-command replacement is that Cinnamon lets you add custom keyboard shortcuts so I have F5 set to run a version of my custom dmenu setup. This gives me convenient access to ssh to other hosts (what I really care about) as well as a bunch of other things. F5 is the same key as I use for this on my desktop, although it is somewhat less convenient on the laptop keyboard (time may change that).

To get an equivalent of sshmenu I use this launcher applet in the Cinnamon panel. The launcher doesn't directly do SSH and it doesn't have submenus (as far as I know), but you can create menu entries and specify the command that they run and I already have a script to run SSH to a host inside an xterm. It's not as nice as sshmenu but it'll do.

(There is a Cinnamon SSH applet but it didn't install when I tried it and it's not usefully customizable from what I understand. Sshmenu itself is unsuitable for a number of reasons, including that it appears to have been abandoned and doesn't actually work with Ruby 2.0.)

I've done some other less important Cinnamon customizations:

• F4 starts a (local) xterm, partly because I haven't changed the Cinnamon panel's 'terminal' launcher away from running gnome-terminal.

• I'm using this Shutdown menu applet because I find it more convenient than trying to find everything I want in the main Cinnamon menu (and I don't like to be prompted).

• I have a shell script I run on session startup to do things like set X resources for xterm.

(As far as I can tell, Cinnamon is essentially a variant of Gnome 3 and so inherits a lot of the same pieces and settings as the latter. For example, Cinnamon applets are JavaScript snippets that run inside the Cinnamon shell-equivalent, not standalone programs.)

I prefer xterm to gnome-terminal in general, but the one nice feature g-t has (in a constrained laptop environment) is that it has a direct way to open URLs in text. My current experiment with doing this with xterm is to have F6 run a command that pulls out the current selection and starts a browser on it if it looks URL-like (and otherwise just start a browser on nothing in particular). This still leaves me actually selecting the URL, which is more work on a laptop trackpad than it is on my desktop.

(I would like to find a better replacement for xterm someday but it's hard. Everyone either adds features I don't care about instead of the ones that I really would like or gets important things wrong. Often it's both. Note that I'm very picky about how a terminal program should behave and there are very few new features I actually want.)

Hopefully Cinnamon will stay stable for a few years so that I don't have to revisit this process for a while. Although I suppose I'm now basically okay as long as whatever desktop I settle in allows me to run a command in response to a function key, since dmenu is really what makes the environment work.

Sidebar: disabling cursor blinking in gnome-terminal

Since the last time I wrote about it Gnome has switched to a fourth way of doing this, covered very thoroughly in this highly useful reference:

gsettings set org.gnome.desktop.interface cursor-blink false

This probably has some global effect but I don't care any more.

(It also looks like gnome-terminal may still respect old settings stuffed in various corners, although who knows how and how you set which ones of them now on a new machine.)

While I'm writing this stuff down, the gsettings schema for what the standard terminal program is is org.gnome.desktop.default-applications.terminal. Your guess is as good as mine as to what programs respect this.