The IMAP Sent folder can create some odd issues and confusion

September 28, 2023

A very common and popular feature of IMAP based mail readers (IMAP clients) is that when you send email, they will also send the email to one of your IMAP folders, often called 'Sent'. On the one hand this is a good idea and many people count on it to keep a record of all of their outgoing email, but on the other hand it can give rise to mysterious issues, for at least two broad reasons.

The first reason is that what IMAP clients aren't actually 'sending' the email to your Sent folder, because there is no such operation in (SMTP) email. Instead they're using IMAP commands to file a copy of the message in an IMAP folder, after they successfully submitted it to your SMTP gateway. This means that there are two different operations, two different protocols, and two different servers involved, which means that one can work while the other fails.

If there is a Sent filing failure, one thing that varies is whether or not the Sent folder copy is lost permanently; some IMAP clients may keep a local copy and retry later, while others may toss the sent but un-filed message and shrug. Another thing that varies is how IMAP clients report this to people and how those people will report this to you. This can lead to puzzling problem reports if, for example, someone tells you that they can't send email when your SMTP server logs say that their email is going out fine.

In practice there are a significant collection of reasons that the Sent filing portion may fail. Reasons we have seen include but are not limited to:

  • There's no space left in (or for) the Sent folder.
  • The Sent folder has a stuck file lock on the server, so nothing can be written to it.
  • The IMAP server uses more restrictive TLS settings than the authenticated SMTP server, so a sufficiently old IMAP client can talk to the authenticated SMTP server but fails to do TLS to the IMAP server.
  • The IMAP server is overloaded and unresponsive, so the person's IMAP client gives up and times out. (This might be a general problem with one of our NFS fileservers, as people's IMAP folders live in their NFS mounted home directories.)

In our environment the authenticated SMTP server and the IMAP server are tightly coupled (they're on the same machine and the SMTP server relies on the IMAP server for authentication), but in a more split apart environment one can be running without the other.

(Even with the two services tightly coupled, people could also be in a network environment that lets them talk to our authenticated SMTP server but not to our IMAP server.)

The second broad issue is that what's filed in the Sent folder isn't actually the message that was sent out to the SMTP recipients. It will be a faithful copy of what the IMAP client submitted to your SMTP server (unless the IMAP client has a bug), but the Sent folder copy won't have any changes or additions that your SMTP server made after that. For one obvious example, it won't be DKIM signed. If the sender then later forwards their Sent folder copy of the message to you (or anyone else), you'll have to remember that this is a raw copy before your SMTP processing, not the final product that other people saw.

Since this IMAP client configuration is so common there's nothing we can really do except remember both potential issues when dealing with problem reports, and when talking to people about how 'sending mail' (through our systems) can fail.

PS: You can also have problems the other way, if your SMTP server ever accidentally swallows a message (or the bounce is lost). The person who sent the email will have a copy in their Sent folder (and their IMAP client didn't report any errors, since your SMTP server initially accepted the email), so it looks to them like the email went out successfully.


Comments on this page:

From 193.219.181.219 at 2023-09-29 01:26:55:

(Even with the two services tightly coupled, people could also be in a network environment that lets them talk to our authenticated SMTP server but not to our IMAP server.)

One possible workaround might be to let Dovecot handle the SMTP submission as well – it has a Submission proxy nowadays, which I believe can store the sent messages server-side before forwarding them to your actual SMTP server. (Although that means you'd end up with two copies...)

By jonys at 2023-09-29 12:59:31:

Yup, the Dovecot server (and some others) implements RFC 4468 (BURL), which feeds the message to SMTP right from the IMAP Sent folder, so only one copy is performed between the e-mail client and the two servers. See https://doc.dovecot.org/admin_manual/submission_server/

It doesn't avoid some of the issues mentioned, such as the lack of DKIM signatures, but at least it prevents the two copies from getting out of sync or one going through and the other staying stuck somewhere between the client and the server.

Unfortunately the only client I know of that has BURL capability was Trojitá, which is no longer developed and ceased being usable after QtWebEngine was deprecated.

By jonys at 2023-09-29 13:04:12:

QtWebKit, not QtWebEngine, I got them mixed up. The latter is still supported, although it lags behind Chromium on updates. Oh well, that's the modern web.

By mg@fork.pl at 2023-10-04 08:43:37:

Trojita was the bright star on MUA sky, sad to see it dead.

Anyway - regarding main subject - I've seen thunderbird to submit message to SMTP multiple times because of IMAP timeout (TB shows messagebox about the problem and allows you to choose "retry" - which apparently restarts whole transaction with SMTP submissions as well)... I think I have to check this dovecot submission proxy - although I guess I wouldn't want to go through it for some system account, mass mailings etc. (which doesn't require Sent)

Written on 28 September 2023.
« Uncertainties over adding IP ratelimits to your local DNS resolvers
Understanding the NMH repl command's '-cc me' and '-nocc me' options »

Page tools: View Source, View Normal.
Search:
Login: Password:

Last modified: Thu Sep 28 23:36:57 2023
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.