Understanding IMAP path prefixes in clients and servers

December 31, 2017

Suppose you have some IMAP clients and they talk to an IMAP server which stores mailboxes somewhere in the filesystem under people's home directories (let's call this the IMAP root for a user). One of the complications of talking about where people's mailboxes and folders actually wind up in this environment is that both the clients and the server get to contribute their two cents, but how they manifest is different.

(As a disclaimer, I'm probably abusing IMAP related terminology here in ways that aren't proper and that I'd fix if I actually ever read up on the details of the IMAP protocol and what it calls things.)

To start with, the IMAP protocol has the concept of a hierarchy of folders and mailboxes, rooted at /. This hierarchy is an abstract thing; it's how clients name things to the server (and how they traverse the namespace with operations like LIST and LSUB). The IMAP server may implement this hierarchical namespace however it wants, using whatever internal names for things that it wants to (provided that it can map back and forth between internal names and protocol level ones know by clients and named in the IMAP subscriptions and so on). Even when an IMAP server stores this IMAP protocol namespace in the filesystem, it may or may not use the client names for things. For now, let's assume that our IMAP server does.

Many IMAP clients have in their advanced configuration options an option for something like an 'IMAP Path Prefix' or an 'IMAP server directory', to use the names that iOS and Thunderbird respectively use for this. This is what it sort of sounds like; it basically causes the IMAP client to use this folder (or series of folders) as a prefix on all of the mailbox and folder names it uses, making it into the root of the IMAP namespace instead of /. If you set this in the client to IMail and have a mailbox that you call 'Private' in the client, the actual name of the mailbox in the IMAP protocol is IMail/Private. Your client simply puts the IMail on the front when it's talking to the server and takes it back off when it gets stuff back and presents this to you.

A client that has an IMAP path prefix and uses LIST will normally only ask for listings of things under its path prefix, because that's what you told it to do. What's visible under the true IMAP root is irrelevant to such a client; it will always confine itself to the path prefix. In our filesystem-backed IMAP server, this means that the client is voluntarily confining itself to a subdirectory of wherever the IMAP server stores things in the filesystem and it doesn't care (and won't notice) what's outside of that subdirectory.

On the server side, the IMAP server might be configured (as ours sadly is) to store folders and mailboxes straight under $HOME, or it might be configured to store them starting in a subdirectory, say $HOME/IMAP. This mapping from the IMAP protocol directory hierarchy used by clients to a directory tree somewhere in the filesystem is very much like how a HTTP server maps from URLs to filesystem locations under its document root (although in the case of the IMAP server, there is a different 'IMAP root' for every user). A properly implemented IMAP server doesn't allow clients to escape outside of this IMAP root through clever tricks like asking for '..', although it may be willing to follow symlinks in the filesystem that lead outside of it.

(As far as I know, such symlinks can't be created through the IMAP protocol, so they must be set up by outside means such as the user sshing in to the IMAP server machine and making a symlink by hand. Of course, with fileservers and shared home directories, that can be any of our Linux servers.)

Using an IMAP path prefix in your client is a good thing if the server's IMAP root is, say, $HOME, since there are probably a great many things there that aren't actually mailboxes and mail folders and that will only confuse your client (and complicate its listing of actual interesting mailboxes) if it looks at them by asking for a listing of /, the root of the IMAP namespace. With an IMAP path prefix configured, your client will always look at a subdirectory of $HOME where you'll presumably only have mailboxes and so on.

The IMAP server is basically oblivious to the use of a client side IMAP path prefix and can't exert any control over it. The client never explicitly tells the server 'I'm using this path prefix'; all the server sees is that the client only ever does operations on things with some prefix.

The net result of this is that you can't transparently replace the use of a client side IMAP path prefix with the equivalent server side change in where the IMAP root is. If you start out with a client IMAP path prefix of IMail and a server IMAP root of $HOME, and then change to a server IMAP root of $HOME/IMail, the client will still try to access IMail/Private, the server will translate this to $HOME/IMail/IMail/Private, and things will probably be sad. To make this work, either you need to move things at the Unix filesystem level or people have to change their IMAP clients to take out the IMAP path prefix.

To make this perhaps a little bit clearer, here is a table of the various pieces and the resulting Unix path that gets formed once all the bits have been put together.

Server IMAP root client IMAP prefix Client folder Unix path
$HOME <none> Private $HOME/Private
$HOME <none> IMail/Private $HOME/IMail/Private
$HOME IMail Private $HOME/IMail/Private
$HOME/IMail IMail Private $HOME/IMail/IMail/Private
$HOME/IMail <none> Private $HOME/IMail/Private

For a given server IMAP root, it doesn't matter whether the client forms the (sub)folder name explicitly or through use of a client IMAP path prefix. If you use multiple clients and only some of them are set up with your IMAP path prefix, clients configured with the prefix will see folder names with the prefix stripped off and other clients will see the full (IMAP protocol) folder path; this is the second and third lines of the table.

(If all of your clients respect IMAP subscriptions, the server may not be able to tell whether or not any particular one of them has an IMAP path prefix configured, or if it's just dutifully following the subscriptions (which are of course all inside the IMAP path prefix you have configured on some clients).)

(This is one of the entries I write partly to get all of this straight in my head.)

Comments on this page:

By Sam Hathaway at 2018-01-01 10:47:57:

You all use Dovecot, right? Is there some reason you can't use hidden namespaces to alias old client side IMAP prefixes to the new server side IMAP root? (As described here.) I'm sure there's a good reason why this wouldn't work for your setup, but I'm not seeing it.

By cks at 2018-01-01 14:34:45:

The honest answer is that I hadn't heard about hidden namespaces until now. If we can use this as a relatively transparent way of migrating things, that would be ideal.

(My experience with the Dovecot wiki documentation has generally been one of relative frustration, where I find it hard to find anything or understand it when I do. It probably makes sense to people who really understand IMAP and Dovecot, neither of which is true for me.)

Written on 31 December 2017.
« Some details of ZFS DVAs and what some of their fields store
Is the C runtime and library a legitimate part of the Unix API? »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun Dec 31 01:14:58 2017
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.