Wandering Thoughts archives

2023-09-28

The IMAP Sent folder can create some odd issues and confusion

A very common and popular feature of IMAP based mail readers (IMAP clients) is that when you send email, they will also send the email to one of your IMAP folders, often called 'Sent'. On the one hand this is a good idea and many people count on it to keep a record of all of their outgoing email, but on the other hand it can give rise to mysterious issues, for at least two broad reasons.

The first reason is that what IMAP clients aren't actually 'sending' the email to your Sent folder, because there is no such operation in (SMTP) email. Instead they're using IMAP commands to file a copy of the message in an IMAP folder, after they successfully submitted it to your SMTP gateway. This means that there are two different operations, two different protocols, and two different servers involved, which means that one can work while the other fails.

If there is a Sent filing failure, one thing that varies is whether or not the Sent folder copy is lost permanently; some IMAP clients may keep a local copy and retry later, while others may toss the sent but un-filed message and shrug. Another thing that varies is how IMAP clients report this to people and how those people will report this to you. This can lead to puzzling problem reports if, for example, someone tells you that they can't send email when your SMTP server logs say that their email is going out fine.

In practice there are a significant collection of reasons that the Sent filing portion may fail. Reasons we have seen include but are not limited to:

  • There's no space left in (or for) the Sent folder.
  • The Sent folder has a stuck file lock on the server, so nothing can be written to it.
  • The IMAP server uses more restrictive TLS settings than the authenticated SMTP server, so a sufficiently old IMAP client can talk to the authenticated SMTP server but fails to do TLS to the IMAP server.
  • The IMAP server is overloaded and unresponsive, so the person's IMAP client gives up and times out. (This might be a general problem with one of our NFS fileservers, as people's IMAP folders live in their NFS mounted home directories.)

In our environment the authenticated SMTP server and the IMAP server are tightly coupled (they're on the same machine and the SMTP server relies on the IMAP server for authentication), but in a more split apart environment one can be running without the other.

(Even with the two services tightly coupled, people could also be in a network environment that lets them talk to our authenticated SMTP server but not to our IMAP server.)

The second broad issue is that what's filed in the Sent folder isn't actually the message that was sent out to the SMTP recipients. It will be a faithful copy of what the IMAP client submitted to your SMTP server (unless the IMAP client has a bug), but the Sent folder copy won't have any changes or additions that your SMTP server made after that. For one obvious example, it won't be DKIM signed. If the sender then later forwards their Sent folder copy of the message to you (or anyone else), you'll have to remember that this is a raw copy before your SMTP processing, not the final product that other people saw.

Since this IMAP client configuration is so common there's nothing we can really do except remember both potential issues when dealing with problem reports, and when talking to people about how 'sending mail' (through our systems) can fail.

PS: You can also have problems the other way, if your SMTP server ever accidentally swallows a message (or the bounce is lost). The person who sent the email will have a copy in their Sent folder (and their IMAP client didn't report any errors, since your SMTP server initially accepted the email), so it looks to them like the email went out successfully.

sysadmin/IMAPSentFolderIssues written at 23:36:57; Add Comment


Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.