On Tue, 30 Nov 2004, Michael Haardt wrote:
> I just had an idea how to improve queue run performance on larger queues:
> How about having each queue runner not spawn a single delivery attempt
> at a time, but a fixed number?
Thanks for the idea, but I am not convinced that it would improve
performance much, if at all. After all, how much resource does it take
to open a file (that is already open by another process), attempt to
lock it, find that it is already locked, and so move on? That is what
happens when a queue runner checks a message that is already being
processed.
Actually, what I have written is not the whole truth. Exim creates a
subprocess in which those tests take place. I suppose some resource
could be saved by doing the test before creating the subprocess
> Right now, you can configure the amount of queue runners in total, but
> quite often I see them stepping on each others toes. A single queue
> runner that keeps a fixed number of deliveries running would not attempt
> to deliver a message that is being tried by another queue runner, just
> to find that the message is locked.
I don't think having a queue runner deliver n messages at once makes any
difference - for each message you still have to test whether some other
Exim process is working on it. Meanwhile, you have added considerable
complication to the queue runner code.
> Right now, a queue runner forks a child and listens on a pipe to it.
> The new queue runner had to have an array of pipes to listen to, starting
> a new child when one ends, unless it reached the end of the queue.
> Keeping 20 children running, the queue would be traversed just 1/20th
> compared to 20 queue runners with one child each.
Circumstances have changed a lot since I started work in Exim nearly 10
years ago. The design was for an environment where over 95% of messages
are delivered right away, so queue runners are dealing with only the
problem messages, which are a small percentage. This is still true in
our environment, though the volume of email has increased enormously, so
5% covers a lot more actual messages.
The bottom line is that Exim does not perform particularly well in
environments where the queue regularly gets very large. It was never
designed for this; deliveries from the queue were always intended to be
"exceptions" rather than the norm.
--
Philip Hazel University of Cambridge Computing Service,
ph10@??? Cambridge, England. Phone: +44 1223 334714.
Get the Exim 4 book: http://www.uit.co.uk/exim-book
