Wandering Thoughts archives

2016-04-18

Why your Apache should have mod_status configured somewhere

Recently, our monitoring system alerted us that our central web server wasn't responding. I poked it and indeed, it wasn't responding, but when I looked at the server everything seemed okay and the logs said it was responding to requests (a lot of them, in fact). Then a little bit later monitoring said it was responding again. Then it wasn't responding. Then my attempt to look at a URL from it worked, but only really slowly.

If you're a long-term Apache wrangler, you can probably already guess the cause. You would be correct; what was going on was that our Apache was being hit with so many requests at once that it was running out of worker processes. If it got through enough work in time, it would eventually pick up your request and satisfy it; if it didn't, you timed out. And if you were lucky, maybe you could get a request in during a lull in all the requests and it would be handled right away.

Once we'd identified the overall cause, we needed to know who or what was doing it. Our central web server handles a wide variety of URLs for a lot of people, some of which can get popular from time to time, so there were a lot of options. And nothing stood out in a quick scan of the logs as receiving a wall of requests or the like. Now, I'm sure that we could have done some more careful log analysis to determine the most active URLs and the most active sources over the last hour or half hour or something, but that would have taken time and effort and we still might have missed sometime. Instead I took the brute force approach: I added mod_status to the server's configuration, on a non-standard URL with access restrictions, and then I looked at it. A high volume source IP jumped out right away and did indeed turn out to be our problem.

Apache's mod_status has a bad reputation as an information leak and a security issue, and as a result I think that a lot of people don't enabled it these days. Our example shows why you might want to reconsider that. Mod_status offers information that's fairly hard to get in any other way and that's very useful (or essential) when you need it, and it's definitely possible to enable it securely. Someday you will want to know who or what is bogging down your server (or at least what it's doing right now), and a live display of current requests is just the thing to tell you.

(This should not be surprising; live status is valuable for pretty much anything. Even when this sort of information can be approximated or reconstructed from logs, it takes extra time and effort.)

web/ApacheModStatusSetup written at 00:33:19; Add Comment


Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.