Our revised Dovecot IMAP configuration migration plans (and processes)
Back at the start of January, I wrote up the goals and problems
of our Dovecot IMAP migration, and in
an appendix at the end I outlined what became our initial migration
plans. We would build an entirely new Dovecot server that was set
up with people's IMAP mail folder storage being a subdirectory of
$HOME/mail (call this the IMAP root), and
then we would get people to move to this server one by one. Migration
would require them to change their clients and might require them
(or us) to move files in Unix. Eventually we would tell the remaining
holdouts that we were just going to turn off the old IMAP server
and they had to migrate now.
Initially, the great virtue I saw in this plan was that it was entirely user driven and didn't require us to do anything. The users did everything, could go at their own speed, and were completely responsible for what happened. In an environment where we couldn't count on clients using IMAP subscriptions so we could know what people's mailboxes actually were, things had to be user-driven anyway, and we generally try to stay out of doing per-user things because it doesn't scale; we have a lot of users and not very many people looking after our central systems (including the IMAP server).
As we talked more and more about this, we realized that the central problem with this plan is that everyone had to migrate and this involved the users doing things (often at the Unix level), or getting someone to help them. As mentioned, we have a lot of users, and some of them are quite important (eg, professors) and can't just be abandoned to their fate. There was no way to make this not be disruptive to people. At the same time, most of our users were not causing any problems, which meant that we'd be forcing a lot of people to do disruptive things (on all of their devices, better not miss one) to deal with a problem created by a much smaller number of users.
If this was the only way to deal with things, we might still have
gone ahead with it. But as I sort of alluded to in passing in the
January entry, it's possible to do
this on a per-user basis in Dovecot using a shell script (see the
bottom of MailLocation).
After we talked it over, we decided that this was the way we wanted
to handle the migration to people's IMAP sessions being confined
to a subdirectory of their
$HOME; it would be done on a per-user
basis and we'd directly target high-priority problem cases. The
vast majority of our current users would forever stay un-migrated,
while new users would be set up to be confined to a
subdirectory from the start (ie, using the new IMAP root).
As much as possible, we wanted this migration to be transparent to
users (or at least important ones). That meant that the IMAP mailbox
names as seen by the clients couldn't change, and that meant that
no matter what we were going to have to move files around; there's
no other way for this to be transparent to clients when you change
the IMAP root. Given that, it wasn't important to pick a new IMAP
root that people already used for mailboxes, so we picked
for various reasons (including that calling it this made it clear
what it was for).
Since this plan means that we're moving user mailboxes around at
least some of the time (in order to migrate problem users), knowing
what those mailboxes were became important enough to get us to
hack some mailbox logging into Dovecot.
Having this information has been extremely reassuring. Even when
it just duplicates the information in a user's
file, it also confirms that that information is accurate and complete.
We started out with plans for a two-stage operation for most users,
where we'd first tell them to move all of their IMAP mailboxes under
mail/' in their client (ie,
$HOME/mail) before some deadline,
then at the deadline we'd make
and flip the server setting that made
$HOME/IMAP their IMAP root.
In practice it's turned out to be easier to do the file moving
ourselves, based on both
.subscriptions and the logs, so our
current approach is to just tell various users 'unless you object,
at time X we'll be improving your IMAP client experience by ...'
and then at time X we do everything ourselves. It's been a little
bit surprising how few actual active mailboxes some of these users
have, especially relative to how much of an impact they've been
having on the server.
(This genuinely does improve the IMAP client experience for people,
for obvious reasons. An IMAP client that is scanning all of your
$HOME and maybe opening all the files there
is generally not a responsive client, not if your
$HOME is at all
PS: Although I haven't been writing about it here on Wandering Thoughts until recently, our IMAP situation has been consuming a lot of my attention and time at work. It's turned into a real learning experience in several ways.