The Maildir mail storage format doesn't seem to work well over NFS

October 5, 2022

There's a relatively widespread view that the Maildir mail storage format is the solution to any issue that one is having with traditional mail message storage. In general I feel that Maildir only solves some problems and makes others worse (although that entry was written in the days of HDDs with very limited IOPS). In our specific environment, we've found that Maildir over NFS to our fileservers is actually pretty terrible once you have a large Maildir folder, one with thousands of messages.

Our IMAP server is Dovecot, which normally automatically supports Maildir format folders if you set one up. Our default folder format is old fashioned mbox format folders (with all messages in one file) and most of our users follow that. However, we have had a very few users who set up Maildir format folders, which are stored in their home directory on one of our NFS fileservers, and then put thousands of messages in those Maildir folders. The observed result on our IMAP server is that them looking at this folder tanked the performance of their mail session and could affect the entire machine's performance.

In this problem's most recent incarnation, this was not because of limited IOPS from HDDs; by that point all of our fileservers were using local SSDs, and mirrored at that. ZFS uses non-linear directories, so directory lookup in large directories shouldn't be a problem and anyway our fileservers have lots of memory and high cache hit rates (and even 10,000 files in a directory isn't large by modern standards). We tried a number of things and investigated in a number of areas, but as near as we could guess, the problem may well be the sheer number of NFS operations that are required to scan a directory and look at things in it. Dovecot normally maintains indexes of folders that mean it doesn't have to touch the folder itself, but maybe it doesn't index Maildir folders or maybe the index was frequently being invalidated for some reason.

(This NFS operations issue may also be specific to the underlying storage being ZFS, although we don't have nay particular evidence either way.)

We no longer have this problem, so I'm somewhat relying on my memories and remembered scars from this. In the end we solved it in the most brute force way possible; the login with the Maildirs was an old account for someone who had long since departed, and when we started talking to relevant parties about maybe getting the person to change the mail folders, the relevant professor told us to just remove the account. But ever since then I have had a little mental note to look out for large Maildir format folders, just in case.

(Because other people may be using them somewhere, we can't disable them in Dovecot. Possibly a future version of Dovecot will let us easily instrument things so we can tell if anyone is using them, and then disable Maildir support if there's no use.)

Comments on this page:

By Ian Z (nobrowser@everywhere) at 2022-10-06 21:19:13:

Dovecot does index maildir folders; it has to update the index when new emails arrive, not necessarily immediately but before a client is allowed to view the folder. And here is one thing you didn't spell out in the post which may be of interest: there are two ways to set up deliveries (well three, but two of them are the same, lol).

You can either let the MTA do the delivery just like it's another local folder (if it has maildir support, as Exim has); this happens behind dovecot's back, so dovecot must reindex on the fly when a client tries to access.

OR you can make the MTA pipe the new emails into dovecot's own MDA, in which case the MDA does the index update at delivery time. I "feel" but do not know this should be the more performant method.

The third way, which in this respect is equivalent to the second, is LMTP, which Exim also supports.

By at 2022-10-11 14:57:10:

What are your experiences with Dovecot and mbox mailboxes?

We have been using Dovecot with mbox since about 2003 (0.9.mumble), and its index database files have frequently given us headaches.

Late 1.x Dovecot versions ran fine, but then the problem re-emerged in 2.x: Dovecot will claim an off-by-one file-length error, and rebuild the index, but the transaction hangs on the client with an empty window, and will have to be restarted. My users have gotten used to this, but they are not happy.

The Dovecot people blame mbox flaws, and have basically refused to fix the problem - but keep claimimg mbox support.

Cheerio, Hauke

By cks at 2022-11-02 21:51:25:

This is a belated answer: we haven't noticed this sort of problem with Dovecot indexes, but we also have the problem that our people don't necessarily report problems to us so it's possible that people are just writing off hung IMAP sessions as par for the course with their IMAP client or our IMAP server or both.

Written on 05 October 2022.
« Our unusual traditional /var/mail setup for people's inboxes
My performance intuitions and the complexities of SSD performance »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Wed Oct 5 22:34:20 2022
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.