Wandering Thoughts archives

2017-06-26

Our MTAs should probably be able to create backpressure

Suppose that you have a complicated mail system, as we do, one with several MTAs and machines involved and mail coming in from multiple directions (inside users, inside machines, outside machines, etc). You would like some always-on precautions (such as ratelimits) that will keep you from being overwhelmed by genuine problems while not harming either normal operations or temporary surges. This sounds hard, but I recently realized that there is probably a general mechanism that will work for it.

If you look at it from the right angle, a multi-machine, multi-MTA environment is a bunch of distributed queues. At an abstract level email moves through this system in much the same way that things move through other multi-hop queue-based systems; everything is queues. We have a lot of experience with queues in programming and system design, and to condense things a lot, we have learned the hard way that simple queue-based systems can easily fly to pieces under overload (for example, see this article).

One of the classical ways of protecting queueing systems from explosive failure under overload conditions is backpressure. When one queue is overloaded, it pushes back by no longer accepting new queue entries. This pressure may ripple back immediately through the system, or it may lead to other queues hitting overload later and pushing back. Backpressure works, and it's often considered essential in robust queue-based distributed systems.

You can see the conclusion here: our multi-MTA setup should be able to apply backpressure. When the queue on one machine gets to be abnormally big, it should start deferring all attempted incoming email (with SMTP 4xx responses). When this happens on an edge machine, such as the external MX gateway, this will naturally push back against the ultimate source of the traffic. When it happens on our central mail handling machine it will probably wind up causing queues on the edge machines to start filling up, which will prompt them to put backpressure on incoming messages. If we set the queue sizes reasonably right, we should be able to not block ordinary mail, deal with temporary surges, and automatically slow down real problems (especially sudden bursts of messages).

Or at least that's the theory. In practice there are tradeoffs involved in not holding email yourself. For the external MX gateway, applying backpressure to senders leaves us at the mercy of their policies on retries and message expiry. On the optimistic side, from the sender's viewpoint backpressure is basically the same as graylisting, and given that graylisting is reasonably common it's very likely that senders cope with it these days. For mail submission servers the issues are more complicated, because mail clients may have various issues if we don't accept their email. However we're already sort of prepared to deal with them since we've already deployed some ratelimiting. We could also limit this backpressure to the mail submission machine that handles unauthenticated connections, since today these tend to be from internal machines (users should be switching to our authenticated SMTP submission machine). Internal machines are historically the most likely source of a sudden flood of email that we want to push back against, and they should mostly handle backpressure. Only mostly, though; there are probably some systems that are not actually using mailers and so will probably just drop deferred email on the floor.

(Clearly the solution for those special systems is a special mail submission machine or IP address that's only used by them and by nothing else. This machine would always accept email, no matter how broken.)

On top of these general practical issues, there is the issue that I don't think Exim has any easy way to do things based on the current queue size. Exim has some load limiters, but these are based on system load average and connection counts. Nor does it look like Exim exposes the current queue size as a variable (or condition). One could probably ${run} a command to look at it, but that seems like a hack at best.

MTABackpressureNeed written at 21:53:00; Add Comment


Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.