Wandering Thoughts archives

2022-11-26

Moving our /var/mail to be local on our IMAP server has gone very well

We have been operating our Unix environment for a very long time, and as a result we've accumulated what are now historical curiosities. One of them is that we put everyone's inboxes in a traditional /var/mail setup (including having them in mbox format), although almost all of our people now read their email only over IMAP. For a long time this /var/mail filesystem lived on our normal NFS fileservers, and the IMAP server accessed it over NFS, the same as everyone else (for example, our central mail server). Periodically we observed various things on the IMAP server, such as oddly elevated load averages, and certainly we had at least the perception that IMAP INBOX performance was periodically not great. Eventually we decided to try to improve things by making the IMAP server into another fileserver, with /var/mail local to it.

(The IMAP server has to be a NFS fileserver and export /var/mail because our other systems still need to mount it. We couldn't change that in the switch.)

At this point we've been running this way for eight months, and the result has been very positive. The most important one is that everyone we've asked has reported that their INBOX is as good or better than it was before (and that's our own experience). Our IMAP server now loafs along with a very low load average and our other metrics for its performance are good. Nor have we experienced any operational problems as a result of the switch.

Mechanically, when I say that the IMAP server is now another fileserver I mean that fairly literally; it uses ZFS and the same local software for managing things that our other fileservers do. The physical hardware is different, but the disks aren't; just as our fileservers do, we used 2 TB SATA SSDs (although in this case in a four-way ZFS mirror instead of a two-way one; the server we used has four disk bays, and why not).

Honestly, the clear performance improvement was a little bit startling. Our NFS fileservers and our IMAP server both have 10G-T network links, which should have performance better than a single SSD, although not quite up to two SSDs. Also, our metrics said that the IMAP server's 10G link was very rarely heavily used and almost never saturated (nor did spot observation suggest that the /var/mail fileserver had saturated networking).

My overall feeling about this situation is that it says something about NFS performance in our specific environment, and perhaps in general. Despite fast fileserver disks (in the form of SSDs) and fast networking, something happened in practice. One of the potential corollaries is that we might not get much NFS performance improvement if we migrated to NVMe SSDs (although there's no real chance of that happening any time soon for us).

sysadmin/LocalVarMailImprovement written at 21:24:09; Add Comment


Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.