The goals and problems of our Dovecot IMAP configuration migration
We have a long standing backwards compatibility issue with our
IMAP server, which is that we have it
configured so that the root of the IMAP mail folder storage is
$HOME. Originally this led to Dovecot session hangs, but now it's led to running out of
inodes on the Dovecot server machine and
general NFS load as people's Dovecot sessions rummage all through
their home directories on our fileservers. Today I'm going to talk about
our ideal IMAP configuration, the problems of trying to migrate to
it, and then some thoughts on what we might settle for.
(In other words, our Dovecot configuration currently sets
mail_location to '
If I could wave a magic wand, the Dovecot IMAP configuration we
want is simply one where all mail folders are stored under some
directory in people's home directories, say
$HOME/mail, and IMAP
clients wouldn't need or want an IMAP path prefix. In this world, if you had a mail
folder for private stuff your client would know it as the PrivateStuff
folder and it would be stored in
your IMAP client do a '
LIST "" "*"' operation, Dovecot would only
$HOME/mail and everything under it, not all of
There are three problems with migrating from our current configuration
to this setup. First, there's all of people's current mail folders
that are in places outside of
$HOME/mail, which must be moved
$HOME/mail in order to stay accessible via IMAP. Second,
even for people who have their mail folders in
clients know it under a different IMAP path; right now an existing
$HOME/mail/PrivateStuff would be known to the IMAP client as
mail/PrivateStuff (even if the client hides this from you by having
an IMAP path prefix), where in the new world it would be known as
just PrivateStuff. Finally, some people have their clients set up
with an IMAP path prefix, which would have to be removed to get our
ideal setup (even if their IMAP path prefix is currently
$HOME/mail right now).
There are some ways to improve these. First, if people are willing
to accept extra directories under
$HOME/mail, they can avoid
needing any client changes even if they were already putting all
of their mail folders in a subdirectory; all you do is preserve the
subdirectory structure when you move mail folders around. If you
$HOME/Mail/PrivateStuff, move it to
$HOME/mail/Mail/PrivateStuff instead of
You wind up with the theoretically surplus
(and a 'Mail/' IMAP path component), but all your clients can
continue on as they are.
Second, if a person uses IMAP subscriptions so that the server-stored
subscription information knows all of their mail folders, we can
reliably move all of them into the same hierarchy under
using only server-side information. Server side mail processing in
.procmailrc may need to be updated, however, since
the actual Unix paths will obviously change. Also, use of IMAP
subscriptions (and the IMAP
LSUB command) is far from universal
among IMAP clients (as I've discovered).
As far as I know, Dovecot doesn't provide a way to log information
about what mail folders are SELECT'd, so we can't determine what
actual mail folders exist through tracking client activity; it's
server side IMAP subscriptions or nothing.
So far I've described the configuration that we want, not necessarily
the configuration we're willing to settle for. So what are our
actual minimum goals? While we haven't actively discussed this, I
think what we'd settle for is an end state configuration where IMAP
clients can't search through all of
$HOME or store mail folders
anywhere outside a small set of subdirectories under it. We could
live with a configuration where mail folders could be in any of
$HOME/Mail, and a few others. We can
also live with these being visible in the IMAP mail folder names
that clients use, so that instead of seeing a folder called
PrivateStuff in your client, you either see mail/PrivateStuff (or
Mail/PrivateStuff or so on), or you set an IMAP path prefix in your
client to hide it.
Sam Hathaway's comment on my entry on IMAP paths in clients and servers brought Dovecot's namespaces to my attention, especially the backwards compatibility examples. I don't think these can be used to migrate towards our ideal configuration, but it's possible they could be used to create something like what we're willing to settle for.
(They could also be used to strip out prefixes from the IMAP paths
that clients send us, but in our specific situation I don't think
there's much point in doing that. The hard part is getting people's
mail folders under
$HOME/mail, and we don't really care if their
path there winds up being eg
Sidebar: A brief note on the mechanics of migration
For various reasons, we have no intention of operating a single Dovecot server with different configurations for different users (ie, with some migrated to the new, confined configuration and others using the old one). Instead we'd do the migration by building an entire new IMAP server under a new name with the new configuration, and then telling people what they had to do to switch over to using it. New people would be pointed to the new server (and blocked from using the old one), while existing people would be encouraged and perhaps helped to migrate. Eventually we'd be down to a few stubborn holdouts and then we'd give them no choice by turning the old server off.
Conveniently the current IMAP server is running Ubuntu 14.04, which means that it has a natural remaining lifetime of about a year and a quarter. This is perhaps enough time to actually get everyone migrated without too much pain.
(Then in a year or two more we'd quietly switch back to using the old IMAP server name, because it really is the best name for an IMAP server.)