IMAP servers can have significant imbalances between disk and network IO
Here's something that hadn't really occurred to me until I saw it happen: an IMAP server can have a significant imbalance between the levels of disk IO and network IO (network traffic). This imbalance can go in both ways; you can have a lot of read or write IO with low network traffic, and less commonly you can have a lot of network traffic with low disk IO. I think an imbalanced situation is especially likely with disk writes, because of the nature of the IMAP protocol.
IMAP is deliberately a high level protocol; it gives you abstract
access to your mailboxes. One of the decisions in the protocol is
that it should avoid round-tripping data through the client as much
as possible. For example, to copy messages from one IMAP folder to
another, your client doesn't download them and then upload them to
store them in the new folder; instead, it issues a
COPY command that
tells the IMAP server 'copy these messages to this folder'. If
you're telling the server to copy (or
bunch of messages, your network traffic is low but the server's IO
traffic is potentially high (if they're big messages). The IMAP
protocol is full
of commands where the client asks the server to do a lot of work
on its behalf. Often this will result in a lot of server disk IO
for relatively little network traffic to and from the client.
One situation an imbalance the other way can happen is if the IMAP server has enough RAM to be holding the data that the client wants in disk cache. Servers can hold gigabytes in disk cache, which will typically vastly exceed the bandwidth of clients, and these days individual messages and MIME parts can be gigantic. There are limits to this, since 10G networking is actually pretty fast compared to other speeds today.
(This is from the server perspective, where you're looking at disk IO versus network bandwidth on the server. The IMAP clients probably mostly don't have 10 G connections.)
On the one hand, this is obvious when I write it out like this. On the other hand, it's not how our NFS fileservers behave. On NFS servers, you have to be doing something unusual for network traffic to not be roughly balanced with disk IO traffic, because the two map pretty directly to each other. This isn't the case on an IMAP server.
(These days our IMAP server is also a NFS fileserver for reasons beyond the scope of this entry, which can make it interesting to try to figure out just what is causing performance problems.)