Why your Apache should have mod_status configured somewhere

April 18, 2016

Recently, our monitoring system alerted us that our central web server wasn't responding. I poked it and indeed, it wasn't responding, but when I looked at the server everything seemed okay and the logs said it was responding to requests (a lot of them, in fact). Then a little bit later monitoring said it was responding again. Then it wasn't responding. Then my attempt to look at a URL from it worked, but only really slowly.

If you're a long-term Apache wrangler, you can probably already guess the cause. You would be correct; what was going on was that our Apache was being hit with so many requests at once that it was running out of worker processes. If it got through enough work in time, it would eventually pick up your request and satisfy it; if it didn't, you timed out. And if you were lucky, maybe you could get a request in during a lull in all the requests and it would be handled right away.

Once we'd identified the overall cause, we needed to know who or what was doing it. Our central web server handles a wide variety of URLs for a lot of people, some of which can get popular from time to time, so there were a lot of options. And nothing stood out in a quick scan of the logs as receiving a wall of requests or the like. Now, I'm sure that we could have done some more careful log analysis to determine the most active URLs and the most active sources over the last hour or half hour or something, but that would have taken time and effort and we still might have missed sometime. Instead I took the brute force approach: I added mod_status to the server's configuration, on a non-standard URL with access restrictions, and then I looked at it. A high volume source IP jumped out right away and did indeed turn out to be our problem.

Apache's mod_status has a bad reputation as an information leak and a security issue, and as a result I think that a lot of people don't enabled it these days. Our example shows why you might want to reconsider that. Mod_status offers information that's fairly hard to get in any other way and that's very useful (or essential) when you need it, and it's definitely possible to enable it securely. Someday you will want to know who or what is bogging down your server (or at least what it's doing right now), and a live display of current requests is just the thing to tell you.

(This should not be surprising; live status is valuable for pretty much anything. Even when this sort of information can be approximated or reconstructed from logs, it takes extra time and effort.)


Comments on this page:

By Twirrim at 2016-04-18 01:01:27:

This may seem a little out of left field, but I've found logstalgia (http://logstalgia.io/) really useful with Apache logs on occasion. Here's a youtube clip of it processing logs: https://www.youtube.com/watch?v=SJy0i62dI64

It's easy to process logs coming in over an ssh tunnel, and I've been able to quickly spot interesting traffic patterns that way that have eluded me from quick log analysis.

By billings at 2016-04-19 21:07:39:

We set up our mod_status page for Nagios to check. There's a lot of stuff in it that lets us discover problems before they cause issues with service, as well as generate graphs of utilization, to help us properly identify problems.

Written on 18 April 2016.
« Why Unix needs a standard way to deal with the file durability problem
Today's odd spammer behavior for sender addresses »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Apr 18 00:33:19 2016
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.