How we moved from a black-box mailer configuration to a white-box one

April 26, 2010

One of the problems with our old central mail system is that it was a black-box mailer configuration; the valid local addresses and domains were defined by what the mailer accepted, and what the mailer accepted was more or less in code, not in handy configuration files. This presented an obvious problem for the evolution of our mail system.

So our first job was to reverse engineer what addresses the central mailer accepted, and figure out how to turn these into reusable information for the new mail machines we wanted to build. Once the dust settled, it turned out that this was not too difficult to machine generate, as all local addresses were equally valid on all local host and domain names. The local addresses were (mostly) made up from 'real' accounts in /etc/passwd, system aliases, and simple mailing lists. Some brute force shell scripting was able to generate a merged list of all possible local addresses.

(We 'parsed' the system aliases with some very brute force code that took advantage of the fact that we had adopted a very rigid formatting convention for the system aliases file.)

The local domain list was harder, because a lot of it was basically implicit in the central mailer's configuration. In the end we took the simple approach: we generated our master list of local hosts by scanning our DNS information to look for hosts and domains that were MX'd to our mail handling machines. To avoid having to parse all of the format variations allowed in BIND's DNS zone files, we did DNS zone transfers from our master server for all of our domains (with 'dig axfr'); this got us the data in a format that was consistent and easy to scan. This did not quite get us all of the machines that the central mail system would handle mail for, but it did get us the list of machines that we wanted the central mail machine to handle mail for, so we called this a feature.

(And without an MX, A, or CNAME record, no outside machine should ever be trying to send email for other machines to our mail machines.)

However, the difficult part of this work was not creating the code to generate the lists but reading through the mailer configuration to figure out just what addresses it accepted and how this was controlled. As a black-box mailer, our central mail machine had some dark and frustrating corners; for example, it turned out that it thought it should handle all email for machines in our domains except for a specific exemption list of machines.

(This was a sensible decision way back when, when there were no other mailers inside our domains and this saved maintaining a long list of machines and domains, but became less and less good over time as more and more machines cropped up that wanted to handle their own email.)

Written on 26 April 2010.
« An observation about Twitter (and Google)
Evolving our mail system step 1: adding an external mail gateway »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Apr 26 01:09:47 2010
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.