Why Exim has a single queue for all email
July 6, 2012
In a recent entry I mentioned that Exim puts all pending email into a single queue instead of having multiple queues (with one per destination domain or the like). Given that multiple queues make it much easier to insure that a single slow destination doesn't affect email to other places, you might wonder why Exim has made such a choice. The answer is that a single queue is effectively forced on Exim because of a core design decision.
All mailers have two conceptual stages for processing a message: they take the initial top level addresses and map them into destinations, then deliver the email to the various destinations. Most mailers do this mapping process once and then save all of the destination addresses, but Exim instead repeats the mapping process every time it retries a message. And this is what makes the difference.
When you do this mapping process only once, it's easy to have per-destination queues; you determine the destinations at the start, add the message to the appropriate queues, and you're done; you can then go through each queue separately. When the mailer is redoing the mappings every time (and the results can change), what you generally wind up with is not queues but limits on how many simultaneous deliveries you can have to any single destination. You can implement this, but it requires much more coordination and complexity than simply starting N delivery processes for a particular queue.
(You wind up with a situation where each delivery process has to check in with something every time it wants to deliver to a new destination.)
For a mailer that redoes mappings every time, the straightforward approach is what Exim has: a single queue of messages and completely independent delivery processes (with per-message locking). The downside of this is that you can wind up with a bunch of your delivery processes all trying to deliver to the same unresponsive destination.
* * *
Atom feeds are available; see the bottom of most pages.