Re: [exim] Should queue processing be rewritten in Exim?

> Other MTA's -can- manage large queues, and for what i know, without
> storing them in ram with the risk of losing data. Stop focussing on
> ramdrives and queues in memory, instead, focus on -why- it is Exim is
> slow with large queues.

I followed this thread for a while and - sorry - it is not leading
anywhere. Again, because we had this discussion a few times now.

There are two very different problems in Exim. One is basic queue
operations and the other is handling large queues. Don't confuse them.

The first is not easy to change, and looking at other MTAs is not going
to help much if you want to squeeze out the most. Once Usenet had
the same problem, and the best idea ever to explore new ways of storing
messages was an storage API. Exim not only lacks that, it even lacks the
documentation which operations are required on queue entries. Once it
was documented, the existing code had to be moved into a new module, with
a documented API in between. Then you could explore new ways to implement
queues, and still keep the old storage implementation for production.

The second is way easier, if all you ask for is running a queue with
e.g. 100,000 entries. Right now, Exim requires a very nonlinear effort
for large queues, because each queue runner scans the entire queue, or
at least a part of it, and conflicts with other queue runners. Using a
single queue scan and forking deliveries off that is easy and gives way
more performance. Unfortunately, you can't say how many deliveries will
run, as a single delivery may start yet more children, but in the real
world, it works well enough to forget the whole problem for a while.

Michael