Here is an important thing about using magic SysRq in practice that I recently learned the hard way:

If you're about to use magic SysRq to diagnose a problem, start by using it to increase the log level.

Specifically, invoke magic SysRq and use 9 to set kernel console logging verbosity to maximum. What this does is insure that all kernel messages go to the console, including all messages produced by magic SysRq operations.

(I believe you can also do this through dmesg with 'dmesg -n debug', which will be most useful if done in advance, say at boot time. Your distribution may have some magic settings file that controls this for you and for that matter it can probably be set as a kernel command line parameter. And of course all of your servers should have magic SysRq enabled.)

If you are naive like the me of a few days ago, you might innocently believe that of course magic SysRq messages will always be shown on the console because it would be kind of stupid to do otherwise. I am sad to say that this is not the case, at least for serial consoles. At the default kernel log levels for at least Ubuntu 12.04, it's quite possible for information dumps from some useful magic SysRq operations (like 'show memory state' and 'show task state') to not be displayed on the serial console, although they are making it into the kernel log buffer (and may be saved somewhere if your system is not too broken).

Speaking from personal experience, this is very frustrating. It's also dangerous because the situation looks just like a kernel that's in so much trouble that various magic SysRq operations aren't completing; as such it can make you more inclined to force-reboot a system that doesn't actually need it.

(Discovering this after the fact is also frustrating if you also discover that the system only managed to write partial logs and you've thus missed capturing potentially vital debugging information.)

I maintain that this is a kernel bug, but even if I could manage to convince the kernel developers of that it would be something like half a decade or more before the fix made it into all of the kernels we and other people will be using. In the mean time, remember that the first magic SysRq operation you want to do is 9 for maximum console logging.

PS: if you're using serial consoles see this caution about serial consoles and magic SysRq, although I suppose the situation may have changed in the last six years (I haven't tested since back then).