Re: [exim] Exim 5

On Wed, Feb 07, 2007 at 03:08:04PM +0000, Philip Hazel wrote:
> One might feel that after 10 years a big new re-design is long overdue.
> On the other hand, large computing projects often get bogged down and
> never get done; incremental changes often have a better chance. But as I
> say, it's up to you guys.

I run Exim since autumn 1998 now, having used Qmail before, and tweaked
the code here and there. IMHO, we can take another round of leaving the
major design in place. So, what could we try out now to learn about it,
that could possibly reason a re-design when being understood completely?

During the last decade, both CPU and RAM experienced a significant
speed up, CPU more so than RAM. I/O on the other side improved only
very moderatly. Although RAID is very common by now, _common_ file
systems are unable to make good use of it. Exim, like all free MTAs
I am aware of, uses a naive approach to store its queue. That simply
does not scale any longer.

We saw that situation with INN before. Being faced with the problem,
the developers introduced the storage API, thus isolating work on actual
storage methods from the rest. Following that approach sounds very
useful: Introduce a queue storage API and use the current queue access
code as its first method. Right now, trying to use more directories in
the split_spool configuration requires changes all over Exim. Not good.
Trying to understand all queue operations required by Exim requires to
read way more source than it should. Introducing a new API fits nicely
in the current design, when you think of routers, transports and lookups.

Having reached that, the way is free to try some new ideas. Sun may
show what they can do with ZFS on the transaction layer. We could try
to use sequential volumes, like one of INNs storage methods. Berkeley DB
might be interesting. The API might be used to interface a central queue
manager that merges simultaneous operations like storing multiple mails
with a single I/O transaction, if the incoming rate is high enough.

AFAIK, none of the free MTAs made that step yet, but it's a matter
of time. There are commercial MTAs that promise to do exactly that,
if I can believe the advertisement I saw.

Michael