Our current mail system's configuration

April 7, 2010

A while back I described our old mail system's configuration. Now it's time to describe our current mail system's configuration ('current' as of April 2010, although it's been pretty stable for the past year or two).

Unlike our old mail system, we now trust NFS; we keep /var/mail on our fileservers, along with everything else important, and the mail machines that need to deal with it use NFS. This has significantly simplified things.

The current email system looks like this:

  • mail from the outside world comes in to our MX gateway, where it is run through a spam checking process and then forwarded to our central mail machine.

  • our central mail machine handles all aspects of email to local addresses; it delivers to /var/mail (and to people's 'oldmail', which keeps a copy of all email to them for the 14 days or so), expands user .forwards and local mailing lists, and so on. It normally delivers email directly to the outside world (using a variety of IP addresses); however, we found it necessary to forward spam-tagged email for the outside world to a separate machine for delivery.

    Users are now encouraged to have procmail and so on deliver directly to /var/mail instead of using the old special addresses that we used to use (although those addresses are still supported).

  • the spam-forwarding machine accepts email from the central mail machine and sends it to the outside world.

There is still a separate mail submission machine for outgoing email (whether from user PCs or our servers). As before, it routes email for our domains to the central mail machine and otherwise sends email straight to the outside world.

There is a separate IMAP/POP server; it accesses everything over NFS, with user inboxes in the NFS-mounted /var/mail and user mail folders stored in their home directories. We have not had any problems with NFS locking between the IMAP server and the central mail machine.

That the MX gateway is separate from the central mail machine is an accident of history, but I think that it simplifies the mailer configuration for both of them. It also means that the system is more resilient in the face of NFS fileserver problems. Since the central mail server accesses /var/mail and user home directories, it is entirely dependent on all of our fileservers working; by contrast, the MX gateway is basically indifferent to NFS, since all it does with email is forward it to the central mail server.

(All of these machines have mirrored system disks, because they do have email sitting in their local spool areas while it's in the process of being delivered or shuffled around.)

Comments on this page:

From at 2010-04-07 00:58:14:

I elected not to NFS mount /var/mail from our mail server because I am very nervous about the notion of NFS-mounting, uh, mutually. (Although, actually, this is one place automount softens the blow, to be sure.) So I forced everyone onto POP and IMAP, basically. Not sure I'd make the same call now, but everyone's gotten over it, so what the heck.


From at 2010-04-07 04:20:04:

Out of curiosity, which route did you go down for the mailservers? Exim, Postfix, (dare I suggest it qmail?) or something else?

By cks at 2010-04-07 08:41:50:

We use Exim on all of these mail servers (and Postfix on all of our client machines, like the servers that just send everything to the mail submission machine). I wrote more about this in PostfixVsExim.

From at 2010-04-07 12:11:18:

May I ask what your requirements are? How many users you have, and what level of uptime is expected? Is this the main mail setup for all of utoronto.ca?

I can't really imagine a whole server dedicated to POP/IMAP - the CPU/memory requirements are so low for this task that you'd have to get into hundreds of thousands of users at least, possibly millions before separating the task would be necessary. If we weren't running Spamassassin on our mail server, the CPU usage would be so close to 0 that it would barely be worth monitoring at all. We have about 5000 mail accounts.

The only other reason I can think of is redundancy, but then you wouldn't need a separate machine, you'd just put all services on one server and then cluster the server.


By cks at 2010-04-07 14:48:00:

This is for Computer Science; UTORMail (the mail system for '<user>@utoronto.ca') has a much bigger configuration. In retrospect the machines are somewhat overconfigured for what they do, but we didn't know it at the time we set them up and we like using separate physical machines for different services in order to get service independence (and easier configuration).

From at 2010-04-07 16:40:51:

What IMAP daemon are you running to serve the mailboxes? Is it running on the fileservers, or also mounted via NFS elsewhere? --bda

By cks at 2010-04-07 17:27:15:

We use Dovecot on the IMAP server machine. It's configured to store its cached mailbox indexes on the local disk, because they're completely replaceable and this way is faster, but everything else comes over NFS from the fileservers.

Putting all of our mail-related stuff on our NFS fileservers has been clearly the right thing to do in our environment; it's simplified a bunch of things, resulted in a less tangled set of NFS mounts, and has probably resulted in better performance overall.

(As a crucial core filesystem, we put /var/mail on a highly replicated chunk of storage; it's part of a ZFS pool built on top of a four-way mirror, instead of our usual two-way mirrors.)

Written on 07 April 2010.
« How not to set up IP aliases on Ubuntu (and probably Debian)
A little script: sshup »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Wed Apr 7 00:20:17 2010
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.