Re: [Exim] Performance bottleneck scanning large spools.

Top Page
Delete this message
Reply to this message
Author: Philip Hazel
Date:  
To: Otavio Exel
CC: Exim mailing list
Subject: Re: [Exim] Performance bottleneck scanning large spools.
On Thu, 20 Jan 2000, Otavio Exel wrote:

> Philip Hazel wrote:
>
> > The first queue runner picks up one message, finds no retry
> > information for the host, and so starts to deliver - and gets stuck.
> > The next queue runner picks up another message for the same host,
> > still finds no retry information, and so also gets stuck... etc.
> > Only after one has failed and set the retry information will
> > subsequent ones start skipping. Hmm. This is an unfortunate
> > side-effect of the way Exim works. I am not sure I can do much about
> > it.
>
> couldn't the first queue runner add a temporary entry to the retry db
> that other queue runners would understand as: "retry info under
> construction; skip messages for this domain for now"; this entry should
> have a reasonable TTL in case the first queue runner dies abnormally;


(1) That would be silly in the case when the host is actually reachable.
I fear it would hit performance to have a rule "only one delivery to a
remote host at once" in general, just to cover this abnormal case, which
is what would happen.

(2) It wouldn't be perfect anyway - the delivery process has to route
the address before it finds out the host(s). Another process can get in
beforehand.

(3) The retry database is updated only once per delivery process
deliberately, in order to minimize the time it is held open with an
exclusive writing lock. Another updating at the start (several if there
are several addresses and you want to create this block as quickly as
possible) would slow things up, I fear.

If the problem messages have only one recipient, then you can use the
relatively new feature of a regular expression matcher on the address to
skip messages whose recipient is in the bad domain:

exim -q -Rr '/^((?!problem\.domain).)*$'

But this won't work if the message has more than one recipient, since
the regex will match the other recipient(s).


-- 
Philip Hazel            University of Cambridge Computing Service,
ph10@???      Cambridge, England. Phone: +44 1223 334714.