Maybe I've abused my root privileges today, but it's only for the good of the
humanity and any failed experiments can be easily reversed.
Anyway. I couldn't see any reason that Mailman should hand Exim a separate
copy for every 100 recipients - in fact, why split the recipient list at all
when Exim can theoretically handle an unlimited number? - so I switched off
that chunking. The advantages are that a little bit of space and internal
memory bandwidth are saved when messages aren't duplicated. The mail queue
also becomes less cluttered and queue runs go faster since Exim processes one
message at a time during a queue run, but does many parallel deliveries if a
message has multiple recipients. The drawback is that if delivery for one
recipient is very slow, it blocks retrying of all the other recipients.
And then I noticed and realised one thing. When Exim got a single message copy
to deliver to all the 1477 active, regular subscribers of exim-users, it took
a very long time, 4-7 minutes, before the first remote connection was opened.
The reason is, of course, that Exim routes all addresses first, one address at
a time, and over 1000 serial DNS lookups mean a lot of waiting.
So, to get to the point, would it be at all possible to parallelise the DNS
lookups (and is it worth it)?
Remote routers usually don't generate new addresses. The exception is when
the "self" option kicks in. A router can also pass in that case or decline
after DNS lookup if ignore_target_hosts removes all the hosts found.
Idea:
Introduce a new yield and/or a new address_item list, addr_resolving.
Add a function host_find_bydns_async, which will initiate DNS queries using an
asynchronous resolver library[1] and take care of the results provided via
the callback interface (since there are multiple levels of abstraction, I
think multiple levels of callbacks are needed). Routers can call this
function and return, putting the address on the addr_resolving list.
When there are no more addresses to route (addr_new == NULL),
deliver_message() will run the select() loop needed to process the pending
DNS queries. Either the callback associated with host_find_bydns_async or a
callback in each router will put the results somewhere in each addr_item
structure.
When there are no more pending DNS queries, addr_resolving can be moved to
addr_new, and routing can continue. The process repeats until all addresses
have been routed.
Sounds easy, right? :-) The other option I could think of involves
threading...
[1] After a quick survey I suggest libares from
ftp://athena-dist.mit.edu/pub/ATHENA/ares, because a) it's API resembles the
standard resolver (res_*) library's, and b) it's in Debian :-).
--
Magnus Holmgren holmgren@???
(No Cc of list mail needed, thanks)
