Quoth Dave C. on Thu, May 02, 2002 at 11:51:44 -0400
> When trying to dequeue a very large number of messages, I have found a
> number of things.
>
> One, If I do an exim -q, it does one message at a time. It doesnt
> overload the machine, but it takes forever, not taking any advantage of
> parallelleism. Since a LOT of the time is spent waiting on DNS servers
> to answer or remote SMTP servers to accept and process connections, this
> is incredibly ineffecient..
You could try to set more than one queue runner. For example, start
40/50 of them and let the messages be delivered that way. I found that
it worked quiet well in practice. Of course, it will impcat your box.
> If I run several queue runner processes, it goes slightly faster, but
> the incremental gain seems to be lost due to them stumbling over each
> other ( eg, "another processs is handling this message")
In practice that is just a check operation to see if another exim
process has locked the file. It doesn't take that much time nor
resources.
> I use split_spool, so have thought I could use the content of each spool
> sub-directory, and run a queue runner on each one, but then the load
> runs through the roof, and the queue runners abandon the run at the
> configured max load..
That's because the queu runners do the whole spool, not just parts of
it. Thus you are starting a process it create 64 instances... woops.
> What would be good, is if instead of abandoning the run, exim would
> pause a moment (perhaps a configurable time), recheck the load, and loop
> some number (perhaps a configurable one) of times waiting for the load
> to come down.
No, that would be bad. You could run into a situation where the whole
system is waiting for the rest of the processes to stop checking if they
can run.
> I think this would be the fastest way to get a large queue of messages
> deliverd as quickly as the machine was capable of...
Just check how many queue runners you can safly run, and use that.
Empirical measures -- or a tiny perl script -- should do the trick.
> What does anyone else think about this?
Hope this helps.
--
yann@??? -=*=- www.kierun.org
PGP: www.kierun.org/pgp/key-kierun
PGP: 009D 7287 C4A7 FD4F 1680 06E4 F751 7006 9DE2 6318
IRC: nick kierun, server spod.uk.amiganet.org, channel #sanctus
