2007-12-07
Using Linux's magic SysRq feature
The 'magic SysRq' is a feature of the Linux kernel where you can directly invoke certain kernel commands, regardless of what happens to be going on at user level and often even if the system is fairly broken otherwise. Commands are conventionally described by the key that's used to invoke them.
There are three ways of invoking magic SysRq commands:
- writing the command key to
/proc/sysrq-trigger, for example with 'echo <key> >/proc/sysrq-trigger' in a root shell. - on the physical console with Alt-SysRq-<key>.
- on a serial console by BREAK followed by the key (within five seconds or so).
The first method always works, but the latter two are controlled
by the kernel.sysrq sysctl setting; this defaults to on but is
sometimes deliberately turned off by distributions.
(Red Hat seems to turn it off, but Ubuntu leaves it alone and thus
enabled. Check your /etc/sysctl.conf.)
The magic SysRq keys that I find most useful are:
s |
Try to sync all filesystems. |
u |
Unmount all filesystems or make them read-only. |
b |
Immediately reboot the machine, without syncing. |
h |
Print succinct help (any unknown key will do, but
h is conventional). |
o |
Immediately power off the machine. |
t |
Dump out information about all processes. |
r |
Recover your keyboard from a crashed X server, allowing you to switch console virtual terminals, although this may not do you much good. |
(This is in roughly the frequency that I actually use them. The full list of available commands is in Documentation/sysrq.txt in the kernel source tree.)
A number of sources will give you involved sequences for rebooting a machine through magic SysRq commands. I tend to just sync, unmount, and then reboot; if a machine is damaged enough that I have to reboot it through magic SysRq, it is generally damaged enough that processes can't shut down cleanly if you send them signals.
(The case that's happened to us several times is a server's system disk
going partly read-only and partly inaccessible; many things that weren't
in active enough use to be in the kernel's cache were unavailable,
including shutdown et al. In this situation it is very useful that
echo is a shell builtin.)
A number of magic SysRq commands are basically kernel diagnostic aids,
such as t. These dump out a great deal of information, so they tend
to only be really useful if the system is still intact enough to log
things or you have a serial console. Triggering
such diagnostic dumps without physical access (or bothering to use the
serial console) is one of the uses of /proc/sysrq-trigger.
(If the system is moderately intact but not able to log things,
perhaps because your /var has become read only, you may be able
to fish the dumped information out with dmesg and capture it
with cut and paste or the like. This has happened to us.)
A trivia note: magic SysRq stuff always works on the physical console, whether or not it is a kernel console. It only works on serial ports if they are actual serial consoles.