Re: [exim] Should queue processing be rewritten in Exim?

Top Page
Delete this message
Reply to this message
Author: Tony Finch
Date:  
To: Michael Haardt
CC: exim-users
Subject: Re: [exim] Should queue processing be rewritten in Exim?
On Fri, 4 Jul 2008, Michael Haardt wrote:
>
> There are two very different problems in Exim. One is basic queue
> operations and the other is handling large queues. Don't confuse them.
>
> The first is not easy to change, and looking at other MTAs is not going
> to help much if you want to squeeze out the most. Once Usenet had
> the same problem, and the best idea ever to explore new ways of storing
> messages was an storage API. Exim not only lacks that, it even lacks the
> documentation which operations are required on queue entries.


Exim has part of a queue API, in the spool_in.c and spool_out.c files.
However there are other places in the code that know about the queue
layout so, as you say, it would need some refactoring to clean this up.

> The second is way easier, if all you ask for is running a queue with
> e.g. 100,000 entries. Right now, Exim requires a very nonlinear effort
> for large queues, because each queue runner scans the entire queue, or
> at least a part of it, and conflicts with other queue runners. Using a
> single queue scan and forking deliveries off that is easy and gives way
> more performance. Unfortunately, you can't say how many deliveries will
> run, as a single delivery may start yet more children, but in the real
> world, it works well enough to forget the whole problem for a while.


It's probably worth changing Exim to support multiple concurrent
deliveries from a single queue runner. You have done this with an external
program (link below) but it would make sense to have the functionality
built-in.

http://lists.exim.org/lurker/message/20050623.095927.833cd3e6.en.html

Tony.
--
<fanf@???> <dot@???> http://dotat.at/ ${sg{\N${sg{\
N\}{([^N]*)(.)(.)(.*)}{\$1\$3\$2\$1\$3\n\$2\$3\$4\$3\n\$3\$2\$4}}\
\N}{([^N]*)(.)(.)(.*)}{\$1\$3\$2\$1\$3\n\$2\$3\$4\$3\n\$3\$2\$4}}