Re: [exim] 4096 connection barrier

--On 23 February 2009 16:47:20 +1100 Ted Cooper <eximX0902w@???>
wrote:

> Marc Perkel wrote:
>> Well, I'm not up to 4096 yet but often run 900 on Monday mornings. I'm
>> filtering spam for over 4000 domains and this is the main box. I might
>> have a project coming up processing far more volume that now.
>>
>> This box is fast. I'm offloading spamassassin on several other servers.
>> I'm using a ram disk for the email queue. I might be adding another
>> 12,000 domains. I'm not doing a lot of delays, although I do throw in a
>> few seconds of suspicious connections but no more than 10 seconds total.
>> So although 4096 sounds like a lot if you throw enough email volume on
>> it you can get there.
>
> The 4096 limit might be more aimed at the limits imposed by linux
> itself. There are open file descriptor limits (default limits are 1024
> per process) which you may reach. I suspect there would be some others
> too, but they are all trivial to increase. It's just a matter of
> adjusting them all which is why this would be considered a local tuning
> customisation and not something that would enter the distribution code
> base.
>
> I know you already use multiple MX records, have you considered
> splitting the MX onto different machines so that you have a fail over?
> Or use round robin DNS to split the traffic between multiple machines?

All that might be doable, but it's a shame to have to deploy more hardware
because of an arbitrary limit on process numbers. In fact, it'd be
downright inefficient (and irresponsible what with global warming, and all).

Marc (or others) may well be using clusters for resilience, but when
failover directs all the traffic to one member, that member needs to be
able to satisfy all the demand.

I sympathise with Marc, because I'm using OSX, which has a hard coded
kernel limit of 2500 processes per machine. It's a problem for me because
of our IMAP process counts, and it really is a headache having to deploy
more machines when I've got plenty of overhead with respect to RAM, disk
I/O, network I/O and so on.

So, Exim's limit isn't reached on my machines, but nevertheless the limit
needs revisiting for those who don't have my problem. Marc's right. The
limit is too low for modern hardware.

--
Ian Eiloart
IT Services, University of Sussex
x3148