You probably need to think about how to handle core dumps on modern Linux servers

April 30, 2018

Once upon a time, life was simple. If and when your programs hit fatal problems, they generally dumped core in their current directory under the name core (sometimes you could make them be core.<PID>). You might or might not ever notice these core files, and some of the time they might not get written at all because of various permissions issues (see the core(5) manpage). Then complications ensued due to things like Apport, ABRT, and systemd-coredump, where an increasing number of Linux distributions have decided to take advantage of the full power of the kernel.core_pattern sysctl to capture core dumps themselves.

(The Ubuntu Apport documentation claims that it's disabled by default on 'stable' releases. This does not appear to be true any more.)

In a perfect world, systems like Apport would capture core dumps from system programs for themselves and arrange that everything else was handled in the traditional way, by writing a core file. Unfortunately this is not a perfect world. In this world, systems like Apport almost always either discard your core files entirely or hide them away where you need special expertise to find them. Under many situations this may not be what you want, in which case you need to think about what you do want and what's the best way to get it.

I think that your options break down like this:

  • If you're only running distribution-provided programs, you can opt to leave Apport and its kin intact. Intercepting and magically handling core dumps from standard programs is their bread and butter, and the result will probably give you the smoothest way to file bug reports with your distribution. Since you're not running your own programs, you don't care about how Apport (doesn't) handle core dumps for non-system programs.

  • Disable any such system and set kernel.core_pattern to something useful; I like 'core.%u.%p'. If the system only runs your services, with no users having access to it, you might want to have all core dumps written to some central directory that you monitor; otherwise, you probably want to set it so that core dumps go in the process's current directory.

    The drawback of this straightforward approach is that you'll fail to capture core dumps from some processes.

  • Set up your own program to capture core dumps and save them somewhere. The advantage of such a program is that you can capture core dumps under more circumstances and also that you can immediately trigger alerting and other things if particular programs or processes die. You could even identify when you have a core dump for a system program and pass the core dump on to Apport, systemd-coredump, or whatever the distribution's native system is.

    One drawback of this is that if you're not careful, your core dump handler can hang your system.

If you have general people running things on your servers and those things may run into segfaults and otherwise dump core, it's my view that you probably want to do the middle option of just having them write traditional core files to the current directory. People doing development tend to like having core files for debugging, and this option is likely to be a lot easier than trying to educate everyone on how to extract core dumps from the depths of the system (if this is even possible; it's theoretically possible with systemd at least).

Up until now we've just passively accepted the default of Apport on our Ubuntu 16.04 systems, but now that we're considering what we want to change for Ubuntu 18.04 and I've been reminded of this whole issue by Julia Evans' How to get a core dump for a segfault on Linux (where she ran into the Apport issue), I think we want to change things to the traditional 'write a core file' setup (which is how it was in Ubuntu 14.04).

(Also, Apport has had its share of security issues over the years, eg 1, 2.)

PS: Since systemd now wants to handle core dumps, I suspect that this is going to be an issue in more and more Linux distributions. Or maybe everyone is going to make sure that that part of systemd doesn't get turned on.


Comments on this page:

From 78.58.206.110 at 2018-05-01 06:35:37:

Extraction via coredumpctl should more or less work these days; in situations where it doesn't work, there's /var/lib/systemd/coredump/*.lz4 for newer systems, and journalctl and Python for older ones (whoever thought that putting massive cores in the system log was a good idea)...

I'm not really a fan of Apport/ABRT, but they do have one useful feature. Before systemd took over, my desktop had a custom core handler which would pop up a backtrace via gxmessage – was surprisingly useful, among other things for informing me that something did in fact just crash.

The traditional approach of writing a core file and logging to dmesg may be simple, but it also means a service might be segfaulting for weeks and you'd never notice that.

The nice thing about ABRT, systemd-coredump etc. is they capture some extra metadata including /proc/pid/maps.

In the case of systemd-coredump, this is undocumented and hidden, you have to use something like `journalctl -o verbose COREDUMP_SIGNAL_NAME=SIGSEGV`. :(.

The traditional approach of writing a core file and logging to dmesg may be simple

By logging to dmesg, I assume you mean the arch-specific message Chris blogged earlier. I didn't see those for SIGBUS / "bus error"... and pretty sure not SIGILL nor SIGFPE either.

https://utcc.utoronto.ca/~cks/space/blog/linux/KernelSegfaultMessageMeaning

(This is in my head from working on an intermittent SIGBUS, that turned out to be a kernel bug. https://bugzilla.redhat.com/show_bug.cgi?id=1553979#c24)

By cks at 2018-05-03 14:24:01:

I think that Apport/ABRT/etc make a lot of sense for many desktops, where people only run system programs and they'll get prompted to automatically file a bug report when one of them crashes. But on servers, there's a much higher chance that neither condition is true; many crashes may not be in system programs, and the core dump handler may be effectively silently swallowing them.

(Some of this may be familiarity and if you're going to dedicate yourself to going all-in on a particular system's approach to handling core dumps. If all your systems handle core dumps through systemd, you can just always look in coredumpctl to see if you have surprise core dumps. If only some do, things get more tangled and harder to keep track of, and there's a higher chance that you'll miss things.)

Written on 30 April 2018.
« Microsoft's Bingbot crawler is on a relative rampage here
An interaction of low ZFS recordsize, compression, and advanced format disks »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Apr 30 21:56:54 2018
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.