The troubles Hotmail have been experiencing has brought what appears to
be terrible inefficiency in Exim's use of the wait-xxx DBM files.
Exim keys these files on hostname.
It surprised me that it doesn't key them on domain name. Keying on
hostname has two implications:
1) The same message ID can end up on the waiting list for multiple
hostnames. When the number of waiting messages is large, the size of
the DBM file can become huge, to the point where Exim processes begin
to time out waiting for their turn to play with it.
2) When Exim connects to a keyed host to deliver mail, it only knows
about the messages waiting for that host. However, if the host is
one of several MX hosts for a given domain, Exim won't necessarily
know about the rest of the hosts waiting for that domain. This means
that there may be messages Exim could have handed over to this host,
but won't.
I'm pretty certain of (1), but I haven't read enough of the code to be
sure of (2). I procede assuming that (2) is correct.
Now I realize that delivery for multiple domains may require connections
to a single host (acting as MX for multiple domains), but I don't see
why this makes keying the wait-xxx DBM files on hostname a requirement.
I can certainly see that there's a trade-off involved, and that the
efficiency of the strategy used depends on the ratio of hosts that host
for multiple domains vs the hosts that are members of MX clusters for
one or more domains.
The way I see it, my Hotmail woes would be largely solved by having Exim
key the wait-xxx DBM files on domain, although Philip's suggestion of
placing the files on a RAMdisk would certainly be a viable system-level
alternative.
Firstly, the size of the DBM file would be drastically reduced. I
predict that my 5-12MB wait-remote_smtp DBM file would shrink to 1-2MB.
This would reduce the amount of time spent waiting for a lock on the
file.
Secondly, Exim would make better use of the few Hotmail connections it
manages to establish, because it would know about _all_ messages waiting
for the domain, instead of just those messages waiting for the host to
which a connection had been established.
So what I need to know is whether the acid still hasn't worn off and
I've completely misunderstood how the wait-xxx DBM files are used, or
whether it's worth my trying to hack up Exim to key them on domain
instead of host.
