IMAP servers can have significant imbalances between disk and network IO

July 14, 2022

Here's something that hadn't really occurred to me until I saw it happen: an IMAP server can have a significant imbalance between the levels of disk IO and network IO (network traffic). This imbalance can go in both ways; you can have a lot of read or write IO with low network traffic, and less commonly you can have a lot of network traffic with low disk IO. I think an imbalanced situation is especially likely with disk writes, because of the nature of the IMAP protocol.

IMAP is deliberately a high level protocol; it gives you abstract access to your mailboxes. One of the decisions in the protocol is that it should avoid round-tripping data through the client as much as possible. For example, to copy messages from one IMAP folder to another, your client doesn't download them and then upload them to store them in the new folder; instead, it issues a COPY command that tells the IMAP server 'copy these messages to this folder'. If you're telling the server to copy (or MOVE) a bunch of messages, your network traffic is low but the server's IO traffic is potentially high (if they're big messages). The IMAP protocol is full of commands where the client asks the server to do a lot of work on its behalf. Often this will result in a lot of server disk IO for relatively little network traffic to and from the client.

One situation an imbalance the other way can happen is if the IMAP server has enough RAM to be holding the data that the client wants in disk cache. Servers can hold gigabytes in disk cache, which will typically vastly exceed the bandwidth of clients, and these days individual messages and MIME parts can be gigantic. There are limits to this, since 10G networking is actually pretty fast compared to other speeds today.

(This is from the server perspective, where you're looking at disk IO versus network bandwidth on the server. The IMAP clients probably mostly don't have 10 G connections.)

On the one hand, this is obvious when I write it out like this. On the other hand, it's not how our NFS fileservers behave. On NFS servers, you have to be doing something unusual for network traffic to not be roughly balanced with disk IO traffic, because the two map pretty directly to each other. This isn't the case on an IMAP server.

(These days our IMAP server is also a NFS fileserver for reasons beyond the scope of this entry, which can make it interesting to try to figure out just what is causing performance problems.)

Written on 14 July 2022.
« How Unix didn't used to support '#!', a brief history
'iptables -L' doesn't show you interface matches on rules by default »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Thu Jul 14 22:59:33 2022
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.