Re: [exim] Fastest Exim server ever

On Feb 29, 2008, at 1:52 PM, Ken Price wrote:

>> The MTA has to be able to receive 50k/second in bursts and get
>> them out
>> as quick as possible. Probably not as fast as coming in but from
>> what I
>> understand delivery speed is important.
>> It's coming from some sort of SQL server that they say can deliver it
>> that fast.
>
> Don't think one server. Think multiple, less powerful servers. 50k
> unique messages a second isn't a realistic goal from a single server.
> Theoretically, you're talking 50k outbound connections to remote MX
> servers.
>
> Also, and I hate saying this, but the advanced features, flexibility,
> and functionality that make Exim so great will also hinder you. You
> don't need a full blown MTA here. Don't hate me, but I REALLY
> recommend Qmail in this application. It's fast, lightweight, and for
> this application ... simple. Might as well go with djbdns on the
> local machines for DNS caching, too.


I do a lot of architectural consulting for scalable systems and my
sense is the following:

If they have some app that can lay down 50K emails in one second what
needs to be happening is not SMTP, but something that directly drops
those messages into an MTA's queue directory.

And that MTA should NOT be Exim. Exim is the cat's meow for being
able to PROGRAM your inbound email. A high-performance SMTP delivery
system it is not. Mr Hazel has stated so himself.

And for the reason that Marc mentioned (non-RFC compliant behavior in
regards to MX records) and a few others, it should NOT be Qmail.

The MTA I would recommend for high speed delivery of these messages
is Postfix. A highly tuned Postfix configuration at that.

As for your hardware, you want a server with LOTS of RAM, dual quad
core, local caching name server, and a VERY fast disk subsystem.

If the application really does call for SMTP for initial delivery out
of the App, then the executable that sending these will be making TCP
connections directly out of the app to an STMTP server. And, as Ken
rightly suggests, this calls for multiple servers configured as above
sitting behind a Load Balancer. You need to pipeline as much as
possible out of that app, and that takes tuning.

MY 2 cents.


Brian