== How our IMAP server wound up running out of inodes On Twitter, [[I mentioned that we'd run out of inodes on a server https://twitter.com/thatcks/status/934234166849363969]], and then a few weeks later I [[made a comment about an IMAP feature https://twitter.com/thatcks/status/941402878777884672]]: > I'm coming to really dislike IMAP clients that don't use > subscriptions, even though the consequences for our server are sort of > our own fault. These two tweets are very closely related, and there is a sad story here (since it's sort of our own fault). In the IMAP protocol, there are two ways to get a list of mailboxes and folders that you have; the _LIST_ command and the _LSUB_ command. The difference between the two is that _LSUB_ restricts itself to things that you have _SUBSCRIBE_'d to (another IMAP command), while the _LIST_ command just lists, well, everything that the IMAP server can discover. When the IMAP server is backed by some sort of database, that 'what it can discover' comes from the database engine; when the IMAP server is storing things in the filesystem as a directory hierarchy, that just translates to a directory listing. (For more details, see [[here https://stackoverflow.com/questions/22014353/what-is-the-use-of-subscribe-and-unsubscribe-commands-in-imap4req1]] and [[here https://superuser.com/questions/61585/what-does-it-mean-to-subscribe-to-an-imap-folder]].) Many IMAP clients use IMAP subscriptions both to track what folders they know about and synchronize the list of known folders between clients, since your IMAP subscriptions are remembered by the server and stored there. However, some clients can't be bothered with this; they simply use _LIST_ to ask the IMAP server for absolutely everything (and presumably then show some or all of it to you). Even when your IMAP server is storing mailboxes and folders in the filesystem, the difference between _LIST_ and _LSUB_ is normally not particularly important because the IMAP server is normally using an area that's only for mailboxes, and the only thing normally found there is mailboxes. Then, unfortunately, there's us. Due to [[the ongoing requirements of backwards compatibility IMAPOurCompatibilityPain]], the root of our IMAP server's mailbox storage is people's _$HOME_. It is quite possible for people's _$HOME_ to contain a lot of things that aren't mailboxes and mail folders, at which point the difference between _LIST_ and _LSUB_ becomes very important to us. If a client uses IMAP subscriptions, what else is in _$HOME_ doesn't matter; the client will only try to look through things you've subscribed to, which are presumably actually mailboxes (and limited). But if the client ignores IMAP subscriptions and just uses _LIST_, it winds up trying to look through everything, and then when it finds directories, it recurses down through them in turn. A year and a half ago, [[our problem IMAPOurCompatibilityPain]] was runaway _LIST_ searches that either ran into symlink cycles or escaped into the wider filesystem, hanging Dovecot and hammering [[our fileservers ../solaris/ZFSFileserverSetupII]]. That's basically stopped being a problem. Today's problem is that some people who use these clients have fairly large _$HOME_s, with things like significant version-controlled source trees and datasets with lots of files and subdirectories. Dovecot maintains [[index files https://wiki.dovecot.org/IndexFiles]] in a directory hierarchy for every mailbox and mail folder that it knows about; when a client uses _LIST_ recursively, this translates to 'at least every directory that Dovecot runs across'. We have Dovecot store its indexes on the IMAP server's local mirrored system disks, because that's a lot faster than getting them over NFS. This is how we wound up running out of inodes on our IMAP server. Dovecot was just trying to store too many index files and directories. Discarding people's index data didn't help for long, because of course their clients did it again and recreated it all after a few days. (Our short term brute force solution was to put in a larger set of SSDs and create a partition just for Dovecot's index data, with the number of inodes set to the maximum value. This has managed to keep us out of danger so far.) I suspect that clients doing this unrestricted _LIST_ usage can't be giving the people using them a really good experience, but apparently it's not so terrible that people stop using them. Unfortunately we don't really have any ideas what specific clients are involved, partly because more and more people are using multiple clients across many different devices. (Our long term fix is going to have to be migrating away from [[our backwards compatibility settings IMAPOurCompatibilityPain]], but that's going to be a very slow process and probably a lot of work. Helpfully it can be done fairly easily for people who actually use IMAP subscriptions, but discussing the issues involved is for another entry.) === Sidebar: How many inodes we're talking about At the moment, our most prolific user has over 1.3 million Dovecot index files and directories, with the next two most prolific users have over 730k and 600k respectively (fortunately it falls off fairly rapidly from there). The overall result of this is that our filesystem for storing this Dovecot index data has over 4.6 million inodes used.