Wandering Thoughts archives

The goals and problems of our Dovecot IMAP configuration migration

We have a long standing backwards compatibility issue with our IMAP server, which is that we have it configured so that the root of the IMAP mail folder storage is people's $HOME. Originally this led to Dovecot session hangs, but now it's led to running out of inodes on the Dovecot server machine and general NFS load as people's Dovecot sessions rummage all through their home directories on our fileservers. Today I'm going to talk about our ideal IMAP configuration, the problems of trying to migrate to it, and then some thoughts on what we might settle for.

(In other words, our Dovecot configuration currently sets mail_location to 'mbox:%h:INBOX=/var/mail/%u'.)

If I could wave a magic wand, the Dovecot IMAP configuration we want is simply one where all mail folders are stored under some directory in people's home directories, say $HOME/mail, and IMAP clients wouldn't need or want an IMAP path prefix. In this world, if you had a mail folder for private stuff your client would know it as the PrivateStuff folder and it would be stored in $HOME/mail/PrivateStuff. Should your IMAP client do a 'LIST "" "*"' operation, Dovecot would only traverse through $HOME/mail and everything under it, not all of $HOME.

There are three problems with migrating from our current configuration to this setup. First, there's all of people's current mail folders that are in places outside of $HOME/mail, which must be moved into $HOME/mail in order to stay accessible via IMAP. Second, even for people who have their mail folders in $HOME/mail, their clients know it under a different IMAP path; right now an existing $HOME/mail/PrivateStuff would be known to the IMAP client as mail/PrivateStuff (even if the client hides this from you by having an IMAP path prefix), where in the new world it would be known as just PrivateStuff. Finally, some people have their clients set up with an IMAP path prefix, which would have to be removed to get our ideal setup (even if their IMAP path prefix is currently mail because they're storing everything under $HOME/mail right now).

There are some ways to improve these. First, if people are willing to accept extra directories under $HOME/mail, they can avoid needing any client changes even if they were already putting all of their mail folders in a subdirectory; all you do is preserve the subdirectory structure when you move mail folders around. If you currently have $HOME/Mail/PrivateStuff, move it to $HOME/mail/Mail/PrivateStuff instead of $HOME/mail/PrivateStuff. You wind up with the theoretically surplus $HOME/mail/Mail directory (and a 'Mail/' IMAP path component), but all your clients can continue on as they are.

Second, if a person uses IMAP subscriptions so that the server-stored subscription information knows all of their mail folders, we can reliably move all of them into the same hierarchy under $HOME/mail using only server-side information. Server side mail processing in things like .procmailrc may need to be updated, however, since the actual Unix paths will obviously change. Also, use of IMAP subscriptions (and the IMAP LSUB command) is far from universal among IMAP clients (as I've discovered). As far as I know, Dovecot doesn't provide a way to log information about what mail folders are SELECT'd, so we can't determine what actual mail folders exist through tracking client activity; it's server side IMAP subscriptions or nothing.

So far I've described the configuration that we want, not necessarily the configuration we're willing to settle for. So what are our actual minimum goals? While we haven't actively discussed this, I think what we'd settle for is an end state configuration where IMAP clients can't search through all of $HOME or store mail folders anywhere outside a small set of subdirectories under it. We could live with a configuration where mail folders could be in any of $HOME/mail, $HOME/IMAP, $HOME/Mail, and a few others. We can also live with these being visible in the IMAP mail folder names that clients use, so that instead of seeing a folder called PrivateStuff in your client, you either see mail/PrivateStuff (or Mail/PrivateStuff or so on), or you set an IMAP path prefix in your client to hide it.

Sam Hathaway's comment on my entry on IMAP paths in clients and servers brought Dovecot's namespaces to my attention, especially the backwards compatibility examples. I don't think these can be used to migrate towards our ideal configuration, but it's possible they could be used to create something like what we're willing to settle for.

(They could also be used to strip out prefixes from the IMAP paths that clients send us, but in our specific situation I don't think there's much point in doing that. The hard part is getting people's mail folders under $HOME/mail, and we don't really care if their path there winds up being eg $HOME/mail/Mail/<something>.)

Sidebar: A brief note on the mechanics of migration

For various reasons, we have no intention of operating a single Dovecot server with different configurations for different users (ie, with some migrated to the new, confined configuration and others using the old one). Instead we'd do the migration by building an entire new IMAP server under a new name with the new configuration, and then telling people what they had to do to switch over to using it. New people would be pointed to the new server (and blocked from using the old one), while existing people would be encouraged and perhaps helped to migrate. Eventually we'd be down to a few stubborn holdouts and then we'd give them no choice by turning the old server off.

Conveniently the current IMAP server is running Ubuntu 14.04, which means that it has a natural remaining lifetime of about a year and a quarter. This is perhaps enough time to actually get everyone migrated without too much pain.

(Then in a year or two more we'd quietly switch back to using the old IMAP server name, because it really is the best name for an IMAP server.)