Re: [Exim] Performance bottleneck scanning large spools.

Top Page
Delete this message
Reply to this message
Author: Philip Hazel
Date:  
To: Gary Palmer
CC: Kai Henningsen, exim-users
Subject: Re: [Exim] Performance bottleneck scanning large spools.
On Tue, 11 Jan 2000, Gary Palmer wrote:

> MaxQueueRunSize option for a reason, and that is if you have a queue
> explosion, then all you really want to do is start getting mail out
> the door and reduce the queue size, and the best way of doing that is
> by running a queue runner every minute or so with only a few hundred
> messages (a thousand or two max) to deliver to remote recipients.


That's an interesting idea. Clearly it would be easy to set up a queue-
runner that stopped scanning the directory when it has 1000 messages.
However, the next one, started a minute later, would probably get more
or less the same 1000 messages - but would that really matter? They
would both be doing deliveries. What *would* be a problem would be the
case where the first 1000 messages obtained from the directory scan all
failed to deliver. No queue-runner would then ever see any other
messages.

> Philip, perhaps its possible to have a ``oh fsck'' mode for exim where
> rather than scan the entire queue, you could treat the split spool as
> 62 individual queues and run one directory, then go on to the next?


That is similar to another suggestion of having a queue-runner just look
at one sub-directory. The problem I see there is how you set up these
queue-runners. Unless you are happy to fork 62 processes in parallel
every time, that is. Doing it in one queue-runner, a directory at a
time, as you suggest, is conceptually easier. Thanks for that idea. It
could even be made the standard way that queue-runners operate in a
split spool environment. I have added it to the Wish List.

> Alternatively you could have an option to split the queue up in the
> queue runner itself. It does the normal directory scan, but after <n>
> messages have been read, it forks off a sub-process to handle those
> <n> messages, and carries on reading.


... but then you end up with <m> queue-runners instead of one, where <m>
depends on the length of your queue and isn't controllable. It breaks
the current "one queue-runner delivers one message at a time" semantics.

-- 
Philip Hazel            University of Cambridge Computing Service,
ph10@???      Cambridge, England. Phone: +44 1223 334714.