There's a relatively widespread view that the Maildir mail storage format is the solution to any issue that one is having with traditional mail message storage. In general I feel that Maildir only solves some problems and makes others worse (although that entry was written in the days of HDDs with very limited IOPS). In our specific environment, we've found that Maildir over NFS to our fileservers is actually pretty terrible once you have a large Maildir folder, one with thousands of messages.

Our IMAP server is Dovecot, which normally automatically supports Maildir format folders if you set one up. Our default folder format is old fashioned mbox format folders (with all messages in one file) and most of our users follow that. However, we have had a very few users who set up Maildir format folders, which are stored in their home directory on one of our NFS fileservers, and then put thousands of messages in those Maildir folders. The observed result on our IMAP server is that them looking at this folder tanked the performance of their mail session and could affect the entire machine's performance.

In this problem's most recent incarnation, this was not because of limited IOPS from HDDs; by that point all of our fileservers were using local SSDs, and mirrored at that. ZFS uses non-linear directories, so directory lookup in large directories shouldn't be a problem and anyway our fileservers have lots of memory and high cache hit rates (and even 10,000 files in a directory isn't large by modern standards). We tried a number of things and investigated in a number of areas, but as near as we could guess, the problem may well be the sheer number of NFS operations that are required to scan a directory and look at things in it. Dovecot normally maintains indexes of folders that mean it doesn't have to touch the folder itself, but maybe it doesn't index Maildir folders or maybe the index was frequently being invalidated for some reason.

(This NFS operations issue may also be specific to the underlying storage being ZFS, although we don't have nay particular evidence either way.)

We no longer have this problem, so I'm somewhat relying on my memories and remembered scars from this. In the end we solved it in the most brute force way possible; the login with the Maildirs was an old account for someone who had long since departed, and when we started talking to relevant parties about maybe getting the person to change the mail folders, the relevant professor told us to just remove the account. But ever since then I have had a little mental note to look out for large Maildir format folders, just in case.

(Because other people may be using them somewhere, we can't disable them in Dovecot. Possibly a future version of Dovecot will let us easily instrument things so we can tell if anyone is using them, and then disable Maildir support if there's no use.)