On Wed, Feb 03, 2016 at 08:33:06AM +0100, Heiko Schlittermann wrote:
> Does anybody know anything about benchmarking an MTA?
> What do we count as performance?
>
> e.g.:
> visible by the user: time, a message spent in the queue
> visible by the admin: (spooled messages)/time the MTA can send
> visible by the sending mta: messages/time the MTA can accept¹
>
> Any other suggestions?
>
> Would anybody be willing to share performance stats?
> (In a first step: submit the results from a tailored eximstats output?)
Meaningful numbers are very difficult to come by because performance
is substantially limited by how much anti-spam/anti-virus scanning
one enables inbound, and which RBLs one queries, ... Or what kind
of rate limits one's outbound traffic is subjected to by the major
mailbox providers (and where most of one's email is sent).
A lot can also depend on proper tuning. The available resources
of an MTA can range by 1, 2 or more orders of magnitude in RAM
capacity, disk IOPS and network bandwidth. Some tuning may be
required to adjust the MTA to utilize the available resources.
So performance is extremely variable from site to site for the same
software. My impression is that Exim is chosen more for flexibility
of built-in content inspection, and not so much performance. I
would, for example, expect Postfix (when not throttled on the
receiving end) to out-perform Exim on raw throughput, but definitely
not on flexibility of ACLs and content inspection.
So, bottom-line, I am not sure how relevant performance benchmarks
really are. Each site needs to do their own measurements if
performance is sufficiently important to them.
As for what to measure, I've usually focused on latency first
because the MTAs I managed were handling people-to-people mail
primarily, rather than bulk newsletters and the like.
Maintaining throughput under load is another important metric, some
MTAs (notably Sendmail) tended to over-throttle under load and do
very poorly when stressed.
--
Viktor.
