MTA Comparison on performance

How to evaluate

A mailing list which contains 341 e-mail addresses was created on the machine for evaluation. These addresses were collected from various domains and messages sent to these addresses were dropped into /dev/null like nobody@domain. Every domain part has different MX RRs.

A message of 2Kbytes were sent to the mailing list.

Traffic on network segment to which the machine is connected was monitored with 'tcpdump' between start and finish time of one delivery through the mailing list.

Graphs were plotted with data from 'tcpdump'. There are two types of graph. One shows number of packets on DNS query and answer. The other shows number of SYN packets sent for SMTP connection request.

Notice

It is not so easy to construct pure internet simulated environment for evaluations, so all evaluations were done in real internet environment. As you know, it is better to measure on different MTA implementations at the same time for more precious comparison, but measurements were done in sequence because of lack of equipments. So, it is impossible to reproduce the same environment for each evaluation, it is, therefore, risky to discuss on detailed differences of MTAs. Only rough tendency should be realized. In addition, I tried but failed to set concurrency parameters to 50 for each MTAs (199 for zmailer). I'd like to re-try evaluations with 50s of concurrency for all MTAs.

How to read the graphs

Sorry for too large graphic images for users who uses small displays.
Zero point of X-axis is the time when first packet (DNS query) was sent out.
Delivery was finished at the time where each line ends in graphs.
Some implementations cause low DNS traffic, because they are utilize additional information in DNS answers effectively.
To compare trends of increase on number of SMTP connections at start-up time, linear lines with same slope also are drown. You can see number of messages to be delivered in a second from the slope.
Number of SMTP SYN is bigger than number of SMTP FIN because some of destination MTAs did not respond.
You can find that some implementations complete DNS query before message delivery, and the others still sent DNS queries among message delivery.
The line which shows number of SYNs for SMTP on SMTPfeed can be seen like stairs. This was a problem (loss of burst SYN packet) of earlier versions of SMTPfeed (SMTPfeed-0.99 or earlier).

Evaluation-1

DNS cache is cleared before evaluation.
(evaluation period: 1999/3/22 12:05-13:39)

Comparison:

Evaluation-2

DNS cache is NOT cleared before evaluation (Fully cached).
(Slow implementations are omitted)
(evaluation period: 1999/3/26 00:08-00:30)

Comparison:

Evaluation-3 (take the Evaluation-2 again)

(evaluation period: 1999/3/27 02:10-02:32)

Comparison:

Environment

FreeBSD-2.2.8-RELEASE
AMD K6/300Mhz
64MB Main memory
2GB HDD IDE
DEC 21140A FastEthernet NIC

Consideration

It is important to utilize additional information included in DNS answer.
It is much effective that all MX/A RRs be cached in local nameserver before deliveries.
Qmail is fast. But it does not any piggybacking which does not reduce SMTP traffic.
Number of connections for SMTP much differs by implementation -- because condition of the Internet had been changed?

Acknowledgements

I'd like to thank you to who offered 'nobody@domain' addresses for this evaluation.