Plan for manual emergency blocks for your overall mail system
Last year, I wrote about how your overall anti-spam system should have manual emergency blocks. At the time I was only thinking about incoming spam, but after some recent experiences here, let me extend that and say that all entry points into your overall mail system should have emergency manual blocks. This isn't just about spam or bad mail from the outside, or preventing outgoing spam, although those are important things. It's also because sometimes systems just freak out and explode, and when this happens your mail system can get deluged as a result. Perhaps a monitoring system starts screaming in email, sending thousands of messages over a short span of time. Perhaps someone notices that a server isn't running a mailer and starts it, only to unleash several months worth of queued email alerts from said server. Perhaps some outside website's notification system malfunctions and deluges some of your users (or many of them) with thousands of messages.
(There are even innocent cases. Email between some active upstream email source (GMail, your organization's central email system, etc) and your systems might have clogged up, and now that the clog has been cleared the upstream is trying to unload all of that queued email on you as fast as it can. You may want some mechanisms in place to let you slow down that incoming flood once you notice it.)
We now have an initial set of blocks, but I'm not convinced that they're exactly what you should have; our current blocks are partly a reaction to the specific incidents that happened to us and partly guesswork about what we might want in the future. Since anticipating the exact form of future explosions is somewhat challenging, our guesswork is probably going to be incomplete and imperfect. Still, it beats nothing and there's value in being able to stop a repeat incident.
(Our view is that we've built are some reasonably workable but crude tools for emergency use, tools that will probably require on the spot adjustment if and when we have to turn them on. We haven't tried to build reliable, always-on mechanisms similar to our anti-spam internal ratelimits.)
We have a reasonably complicated mail system with multiple machines running MTAs; there's our inbound MX gateway, two submission servers, a central mail processing server, and some other bits and pieces. One of the non-technical things we've done in the fallout from the recent incidents is to collect in one spot the information about what you can do on each of them to block email in various ways. Hopefully we will keep this document updated in the future, too.
(You may laugh, but previously the information was so dispersed that I actually forgot that one blocking mechanism already existed until I started doing the research to write all of this up in one place. This can happen naturally if you develop things piecemeal over time, as we did.)
Sidebar: Emergency tools versus routine mechanisms
Some people would take this as a sign that we should have always-on mechanisms (such as ratelimits) that are designed to automatically keep our mail system from being overwhelmed no matter what happens. My own view is that designing such mechanisms can be pretty hard unless you're willing to accept ones that are set so low that they have a real impact in normal operation if you experience a temporary surge.
Actually, not necessarily (now that I really think about it). It is in our environment, but that's due to the multi-machine nature of our environment combined with some of our design priorities and probably some missing features in Exim. But that's another entry.
How I'm currently handling the mailing lists I read
I recently mentioned that I was going to keep filtering aside email from the few mailing lists that I'm subscribed to, instead of returning it to being routed straight into my inbox. While I've kept to my decision, I've had to spend some time fiddling around with just how I was implementing it in order to get a system that works for me in practice.
What I did during my vacation (call it the vacation approach) was to use procmail recipes to put each mailing list into a file. I'm already using procmail, and in fact I was already recognizing mailing lists (to insure they didn't get trapped by anti-spam stuff), so this was a simple change:
:0: * ^From somelist-owner@... lists/somelist #V#$DEFAULT
This worked great during my vacation, when I basically didn't want
to pay attention to mailing lists at all, but once I came back to
work I found that filing things away this way made them too annoying
to deal with in my mail environment. Because MH
doesn't deal directly with mbox format files, I needed to go through
a whole dance with
inc and then rescanning my inbox and various
other things. It was clear that this wasn't the right way to go.
If I wanted it to be convenient to read this email (and I did),
incoming mailing list messages had to wind up in MH folders.
Fortunately, procmail can do this if you specify '
as the destination (the '
/.' is the magic). So:
:0: * ^From somelist-owner@... /u/cks/Mail/inbox/somelist/.
(This is not quite a complete implementation, because it doesn't
do things like update MH's
unseen sequence for the folder. If you
want these things, you need to pipe messages to
In my case, I actually prefer not having an
unseen sequence be
maintained for these folders for various reasons.)
The procmail stuff worked, but I rapidly found that I wanted some
way to know which of these mailing list folders actually had pending
messages in them. So I wrote a little command which I'm calling
mlists'. It goes through my
.procmailrc to find all of the MH
destinations, then uses
ls to count how many message files there
are and reports the whole thing as:
:; mlists +inbox/somelist: 3 :;
If there's enough accumulated messages to make looking at the folder worthwhile, I can then apply standard MH tools to do so (either from home with standard command line MH commands, or with exmh at work).
It's early days with this setup, but so far I feel satisfied. The
filtering and filing stuff works and the information
provides is enough to be useful but sufficiently minimal to push
me away from actually looking at the mailing lists for much of the
time, which is basically the result that I want.
PS: there's probably a way to assemble standard MH commands to
give you a count of how many messages are in a folder. I used
because I couldn't be bothered to read through MH manpages to work
out the MH way of doing it, and MH's simple storage format makes
this kind of thing easy.