Viktor Dukhovni <exim-users@???> (Mi 03 Feb 2016 09:20:49 CET):
> On Wed, Feb 03, 2016 at 08:33:06AM +0100, Heiko Schlittermann wrote:
>
> > Does anybody know anything about benchmarking an MTA?
> > What do we count as performance?
> >
> > e.g.:
> > visible by the user: time, a message spent in the queue
> > visible by the admin: (spooled messages)/time the MTA can send
> > visible by the sending mta: messages/time the MTA can accept¹
> >
> > Any other suggestions?
> >
> > Would anybody be willing to share performance stats?
> > (In a first step: submit the results from a tailored eximstats output?)
>
> Meaningful numbers are very difficult to come by because performance
> is substantially limited by how much anti-spam/anti-virus scanning
> one enables inbound, and which RBLs one queries, ... Or what kind
> of rate limits one's outbound traffic is subjected to by the major
> mailbox providers (and where most of one's email is sent).
I'd like to see a performance that is solely bound to resources we can't
influence. But as - e.g. - Exim's queue runners step on each other's
feet, it't not the fault of the remote site.
> A lot can also depend on proper tuning. The available resources
> of an MTA can range by 1, 2 or more orders of magnitude in RAM
> capacity, disk IOPS and network bandwidth. Some tuning may be
> required to adjust the MTA to utilize the available resources.
Same as above. Of course, lots of RAM accelerate the processing,
lowering the risk of locking problems (given a constant number of queue
runners).
> So performance is extremely variable from site to site for the same
> software. My impression is that Exim is chosen more for flexibility
> of built-in content inspection, and not so much performance.
Exim as a MTA framework :)
> would, for example, expect Postfix (when not throttled on the
> receiving end) to out-perform Exim on raw throughput, but definitely
> not on flexibility of ACLs and content inspection.
Lego vs. Playmobil :)
Yes, I agree, Exim is not famous for it performance, compared with
Postfix. But I'm looking for a better definition of the term
"performance", what do people consider as the number to compare.
> As for what to measure, I've usually focused on latency first
> because the MTAs I managed were handling people-to-people mail
> primarily, rather than bulk newsletters and the like.
That matches the "queue time" (optionally logged as QT=…), the time a
given message spent in the queue. I agree, it's the most user visible
factor.
> Maintaining throughput under load is another important metric, some
> MTAs (notably Sendmail) tended to over-throttle under load and do
> very poorly when stressed.
--
Heiko
