2017-12-28
How our IMAP server wound up running out of inodes
On Twitter, I mentioned that we'd run out of inodes on a server, and then a few weeks later I made a comment about an IMAP feature:
I'm coming to really dislike IMAP clients that don't use subscriptions, even though the consequences for our server are sort of our own fault.
These two tweets are very closely related, and there is a sad story here (since it's sort of our own fault).
In the IMAP protocol, there are two ways to get a list of mailboxes
and folders that you have; the LIST command and the LSUB command.
The difference between the two is that LSUB restricts itself to
things that you have SUBSCRIBE'd to (another IMAP command), while
the LIST command just lists, well, everything that the IMAP server
can discover. When the IMAP server is backed by some sort of database,
that 'what it can discover' comes from the database engine; when
the IMAP server is storing things in the filesystem as a directory
hierarchy, that just translates to a directory listing.
(For more details, see here and here.)
Many IMAP clients use IMAP subscriptions both to track what folders
they know about and synchronize the list of known folders between
clients, since your IMAP subscriptions are remembered by the server
and stored there. However, some clients can't be bothered with this;
they simply use LIST to ask the IMAP server for absolutely
everything (and presumably then show some or all of it to you).
Even when your IMAP server is storing mailboxes and folders in the
filesystem, the difference between LIST and LSUB is normally
not particularly important because the IMAP server is normally using
an area that's only for mailboxes, and the only thing normally found
there is mailboxes. Then, unfortunately, there's us. Due to the
ongoing requirements of backwards compatibility,
the root of our IMAP server's mailbox storage is people's $HOME.
It is quite possible for people's $HOME to contain a lot of things
that aren't mailboxes and mail folders, at which point the difference
between LIST and LSUB becomes very important to us. If a client
uses IMAP subscriptions, what else is in $HOME doesn't matter;
the client will only try to look through things you've subscribed
to, which are presumably actually mailboxes (and limited). But if
the client ignores IMAP subscriptions and just uses LIST, it winds
up trying to look through everything, and then when it finds
directories, it recurses down through them in turn.
A year and a half ago, our problem was
runaway LIST searches that either ran into symlink cycles or
escaped into the wider filesystem, hanging Dovecot and hammering
our fileservers. That's basically
stopped being a problem. Today's problem is that some people who
use these clients have fairly large $HOMEs, with things like
significant version-controlled source trees and datasets with lots
of files and subdirectories. Dovecot maintains index files in a directory hierarchy for
every mailbox and mail folder that it knows about; when a client
uses LIST recursively, this translates to 'at least every directory
that Dovecot runs across'. We have Dovecot store its indexes on the
IMAP server's local mirrored system disks, because that's a lot faster
than getting them over NFS.
This is how we wound up running out of inodes on our IMAP server. Dovecot was just trying to store too many index files and directories. Discarding people's index data didn't help for long, because of course their clients did it again and recreated it all after a few days.
(Our short term brute force solution was to put in a larger set of SSDs and create a partition just for Dovecot's index data, with the number of inodes set to the maximum value. This has managed to keep us out of danger so far.)
I suspect that clients doing this unrestricted LIST usage can't
be giving the people using them a really good experience, but
apparently it's not so terrible that people stop using them.
Unfortunately we don't really have any ideas what specific clients
are involved, partly because more and more people are using multiple
clients across many different devices.
(Our long term fix is going to have to be migrating away from our backwards compatibility settings, but that's going to be a very slow process and probably a lot of work. Helpfully it can be done fairly easily for people who actually use IMAP subscriptions, but discussing the issues involved is for another entry.)
Sidebar: How many inodes we're talking about
At the moment, our most prolific user has over 1.3 million Dovecot index files and directories, with the next two most prolific users have over 730k and 600k respectively (fortunately it falls off fairly rapidly from there). The overall result of this is that our filesystem for storing this Dovecot index data has over 4.6 million inodes used.