Hi,
Some of our mail routing is handled in our MDA for legacy reasons. Mail is
transported to the MDA by a pipe (one recipient per execution of the pipe).
If the recipient address is the final address after any local routing the
message is written to the recipients mailstore. If routing results in new
addresses, the message is piped back into Exim and probably then back to the
MDA again, perhaps a few times in the worst cases.
In the interests of efficiency and also to do better address verification at
SMTP time, I've come up with a variety of routers which emulate what used to be
done in the MDA, and these are working well with one caveat.
In the old scheme two addresses may route to the same mailstore, and the
recipient would receive two copies of the message. Some customers rely on this
behaviour as we add an "X-Envelope-To" header in the MDA which they can filter
on locally to deliver the mail to the correct local recipients at their site.
Using Exim to do the routing, however, causes the duplicate copies of the
message to be discarded. This is documented in the spec.
3.10 | If child addresses are generated, Exim checks to see whether they are
| duplicates of any existing recipient addresses. During this check,
| local parts are treated as case-sensitive. Duplicate addresses are
| discarded.
I enclose the debugging output (-d+route) of a delivery that was set up as
follows:
# /usr/exim/bin/exim -odq -f ollie@??? foo@??? bar@???
Subject: test message
test body
^D
The mail appears in the queue at this stage with two recipients:
0m 280 1CRVc5-000Fcm-45 <ollie@???>
foo@???
bar@???
This is how the addresses route:
# /usr/exim/bin/exim -bt foo@??? bar@???
postmaster@???
<-- foo@???
router = corp_pop, transport = claradeliver_pipe
postmaster@???
<-- bar@???
router = corp_pop, transport = claradeliver_pipe
And this is the output from the delivery run. I've omitted some detail
in the interests of brevity:
# /usr/exim/bin/exim -d+route -M 1CRVc5-000Fcm-45
Exim version 4.43 uid=0 gid=0 pid=61741 D=fbb95cfd
Berkeley DB: Sleepycat Software: Berkeley DB 4.0.14: (November 18, 2001)
Support for:
Lookups: lsearch wildlsearch nwildlsearch iplsearch cdb dbm dbmnz mysql
Authenticators:
Routers: accept dnslookup ipliteral manualroute queryprogram redirect
Transports: appendfile autoreply lmtp pipe smtp
changed uid/gid: forcing real = effective
uid=0 gid=0 pid=61741
auxiliary group list: 0
configuration file is /usr/exim/configure
log selectors = 00002ffc 00016410
trusted user
admin user
skipping ACL configuration - not needed
finduser used cached passwd data for claranet
finduser used cached passwd data for claranet
set_process_info: 61741 delivering specified messages
set_process_info: 61741 delivering 1CRVc5-000Fcm-45
reading spool file 1CRVc5-000Fcm-45-H
user=root uid=0 gid=0 sender=ollie@???
sender_local=0 ident=root
Non-recipients:
Empty Tree
---- End of tree ----
recipients_count=2
body_linecount=1 message_linecount=6
running system filter
rda_interpret (file): /usr/exim/system_filter
expanded: /usr/exim/system_filter
1208 bytes read from /usr/exim/system_filter
data is an Exim filter program
Filter: start of processing
Filter: end of processing
system filter returned 1
Delivery address list:
foo@???
bar@???
locking /usr/exim/spool/db/retry.lockfile
locked /usr/exim/spool/db/retry.lockfile
opened hints database /usr/exim/spool/db/retry: flags=0
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Considering: foo@???
unique = foo@???
dbfn_read: key=R:borked.co.uk
dbfn_read: key=R:foo@???
no domain retry record
no address retry record
foo@???: queued for routing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Considering: bar@???
unique = bar@???
dbfn_read: key=R:borked.co.uk
dbfn_read: key=R:bar@???
no domain retry record
no address retry record
bar@???: queued for routing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
routing bar@???
--------> corp_route_dpb router <--------
local_part=bar domain=borked.co.uk
checking domains
search_open: cdb "/usr/exim/conf/customer_domains.cdb"
cached open
search_find: file="/usr/exim/conf/customer_domains.cdb"
key="borked.co.uk" partial=-1 affix=NULL starflags=0
LRU list:
0/usr/exim/conf/customer_domains.cdb
6/usr/exim/conf/checkbridge_ips
0/usr/exim/conf/ex_domains.cdb
:/usr/exim/conf/mail_excludes
End
internal_search_find: file="/usr/exim/conf/customer_domains.cdb"
type=cdb key="borked.co.uk"
cached data used for lookup of borked.co.uk
in /usr/exim/conf/customer_domains.cdb
lookup yielded:
borked.co.uk in "cdb;/usr/exim/conf/customer_domains.cdb"? yes (matched "cdb;/usr/exim/conf/customer_domains.cdb")
calling corp_route_dpb router
rda_interpret (string): ${lookup mysql{SELECT IF(INSTR(def_pop_box, '@'), def_pop_box, CONCAT(def_pop_box, '@', domain)) FROM clara.domains WHERE domain='${quote_mysql:$domain}'}{$value}fail}
search_open: mysql "NULL"
cached open
search_find: file="NULL"
key="SELECT IF(INSTR(def_pop_box, '@'), def_pop_box, CONCAT(def_pop_box, '@', domain)) FROM clara.domains WHERE domain='borked.co.uk'" partial=-1 affix=NULL starflags=0
LRU list:
0/usr/exim/conf/customer_domains.cdb
6/usr/exim/conf/checkbridge_ips
0/usr/exim/conf/ex_domains.cdb
:/usr/exim/conf/mail_excludes
End
internal_search_find: file="NULL"
type=mysql key="SELECT IF(INSTR(def_pop_box, '@'), def_pop_box, CONCAT(def_pop_box, '@', domain)) FROM clara.domains WHERE domain='borked.co.uk'"
database lookup required for SELECT IF(INSTR(def_pop_box, '@'), def_pop_box, CONCAT(def_pop_box, '@', domain)) FROM clara.domains WHERE domain='borked.co.uk'
MYSQL query: SELECT IF(INSTR(def_pop_box, '@'), def_pop_box, CONCAT(def_pop_box, '@', domain)) FROM clara.domains WHERE domain='borked.co.uk'
MYSQL using cached connection for localhost:3306/clarapw/exim
lookup yielded: postmaster@???
expanded: postmaster@???
file is not a filter file
parse_forward_list: postmaster@???
extract item: postmaster@???
corp_route_dpb router generated postmaster@???
errors_to=NULL transport=NULL
uid=unset gid=unset home=NULL
routed by corp_route_dpb router
envelope to: bar@???
transport: <none>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
routing foo@???
--------> corp_route_dpb router <--------
local_part=foo domain=borked.co.uk
checking domains
search_open: cdb "/usr/exim/conf/customer_domains.cdb"
cached open
search_find: file="/usr/exim/conf/customer_domains.cdb"
key="borked.co.uk" partial=-1 affix=NULL starflags=0
LRU list:
0/usr/exim/conf/customer_domains.cdb
6/usr/exim/conf/checkbridge_ips
0/usr/exim/conf/ex_domains.cdb
:/usr/exim/conf/mail_excludes
End
internal_search_find: file="/usr/exim/conf/customer_domains.cdb"
type=cdb key="borked.co.uk"
cached data used for lookup of borked.co.uk
in /usr/exim/conf/customer_domains.cdb
lookup yielded:
borked.co.uk in "cdb;/usr/exim/conf/customer_domains.cdb"? yes (matched "cdb;/usr/exim/conf/customer_domains.cdb")
calling corp_route_dpb router
rda_interpret (string): ${lookup mysql{SELECT IF(INSTR(def_pop_box, '@'), def_pop_box, CONCAT(def_pop_box, '@', domain)) FROM clara.domains WHERE domain='${quote_mysql:$domain}'}{$value}fail}
search_open: mysql "NULL"
cached open
search_find: file="NULL"
key="SELECT IF(INSTR(def_pop_box, '@'), def_pop_box, CONCAT(def_pop_box, '@', domain)) FROM clara.domains WHERE domain='borked.co.uk'" partial=-1 affix=NULL starflags=0
LRU list:
0/usr/exim/conf/customer_domains.cdb
6/usr/exim/conf/checkbridge_ips
0/usr/exim/conf/ex_domains.cdb
:/usr/exim/conf/mail_excludes
End
internal_search_find: file="NULL"
type=mysql key="SELECT IF(INSTR(def_pop_box, '@'), def_pop_box, CONCAT(def_pop_box, '@', domain)) FROM clara.domains WHERE domain='borked.co.uk'"
cached data used for lookup of SELECT IF(INSTR(def_pop_box, '@'), def_pop_box, CONCAT(def_pop_box, '@', domain)) FROM clara.domains WHERE domain='borked.co.uk'
lookup yielded: postmaster@???
expanded: postmaster@???
file is not a filter file
parse_forward_list: postmaster@???
extract item: postmaster@???
corp_route_dpb router generated postmaster@???
errors_to=NULL transport=NULL
uid=unset gid=unset home=NULL
routed by corp_route_dpb router
envelope to: foo@???
transport: <none>
locking /usr/exim/spool/db/retry.lockfile
locked /usr/exim/spool/db/retry.lockfile
opened hints database /usr/exim/spool/db/retry: flags=0
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Considering: postmaster@???
unique = postmaster@???
dbfn_read: key=R:borked.co.uk
dbfn_read: key=R:postmaster@???
no domain retry record
no address retry record
postmaster@???: queued for routing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Considering: postmaster@???
unique = postmaster@???
postmaster@??? is a duplicate address: discarded
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
routing postmaster@???
--------> corp_pop router <--------
local_part=postmaster domain=borked.co.uk
checking domains
search_open: cdb "/usr/exim/conf/customer_domains.cdb"
cached open
search_find: file="/usr/exim/conf/customer_domains.cdb"
key="borked.co.uk" partial=-1 affix=NULL starflags=0
LRU list:
0/usr/exim/conf/customer_domains.cdb
6/usr/exim/conf/checkbridge_ips
0/usr/exim/conf/ex_domains.cdb
:/usr/exim/conf/mail_excludes
End
internal_search_find: file="/usr/exim/conf/customer_domains.cdb"
type=cdb key="borked.co.uk"
cached data used for lookup of borked.co.uk
in /usr/exim/conf/customer_domains.cdb
lookup yielded:
borked.co.uk in "cdb;/usr/exim/conf/customer_domains.cdb"? yes (matched "cdb;/usr/exim/conf/customer_domains.cdb")
checking "condition"
search_open: mysql "NULL"
cached open
search_find: file="NULL"
key="SELECT 'yes' FROM clara.domain_users WHERE user='postmaster' AND domain='borked.co.uk'" partial=-1 affix=NULL starflags=0
LRU list:
0/usr/exim/conf/customer_domains.cdb
6/usr/exim/conf/checkbridge_ips
0/usr/exim/conf/ex_domains.cdb
:/usr/exim/conf/mail_excludes
End
internal_search_find: file="NULL"
type=mysql key="SELECT 'yes' FROM clara.domain_users WHERE user='postmaster' AND domain='borked.co.uk'"
database lookup required for SELECT 'yes' FROM clara.domain_users WHERE user='postmaster' AND domain='borked.co.uk'
MYSQL query: SELECT 'yes' FROM clara.domain_users WHERE user='postmaster' AND domain='borked.co.uk'
MYSQL using cached connection for localhost:3306/clarapw/exim
lookup yielded: yes
calling corp_pop router
corp_pop router called for postmaster@???
domain = borked.co.uk
set transport claradeliver_pipe
queued for claradeliver_pipe transport: local_part = postmaster
domain = borked.co.uk
errors_to=NULL
domain_data= localpart_data=NULL
routed by corp_pop router
envelope to: postmaster@???
transport: claradeliver_pipe
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
After routing:
Local deliveries:
postmaster@???
Remote deliveries:
Failed addresses:
Deferred addresses:
search_tidyup called
close MYSQL connection: localhost:3306/clarapw/exim
>>>>>>>>>>>>>>>> Local deliveries >>>>>>>>>>>>>>>>
--------> postmaster@??? <--------
locking /usr/exim/spool/db/retry.lockfile
locked /usr/exim/spool/db/retry.lockfile
opened hints database /usr/exim/spool/db/retry: flags=0
dbfn_read: key=T:postmaster@???
no retry record exists
search_tidyup called
changed uid/gid: local delivery to postmaster <postmaster@???> transport=claradeliver_pipe
uid=5000 gid=5000 pid=61743
auxiliary group list: 5000
home=NULL current=/
set_process_info: 61743 delivering 1CRVc5-000Fcm-45 to postmaster using claradeliver_pipe
claradeliver_pipe transport entered
direct command:
argv[0] = /usr/local/bin/claradeliver
argv[1] = ${local_part}@${domain}
direct command after expansion:
argv[0] = /usr/local/bin/claradeliver
argv[1] = postmaster@???
Writing message to pipe
set_process_info: 61745 reading output from |/usr/local/bin/claradeliver ${local_part}@${domain}
writing data block fd=9 size=0 timeout=3600
added header line(s):
X-RCPT: postmaster
X-Envelope-To: foo@???
---
writing data block fd=9 size=420 timeout=3600
writing data block fd=9 size=1 timeout=3600
claradeliver_pipe transport yielded 0
journalling postmaster@???
search_tidyup called
claradeliver_pipe transport returned OK for postmaster@???
post-process postmaster@??? (0)
bar@???: children all complete
postmaster@??? delivered
foo@???: children all complete
LOG: MAIN
=> postmaster <foo@???> R=corp_pop T=claradeliver_pipe
>>>>>>>>>>>>>>>> deliveries are done >>>>>>>>>>>>>>>>
changed uid/gid: post-delivery tidying
uid=5000 gid=5000 pid=61741
auxiliary group list: 5000
set_process_info: 61741 tidying up after delivering 1CRVc5-000Fcm-45
Processing retry items
Succeeded addresses:
postmaster@???: no retry items
foo@???: no retry items
foo@???: no retry items
bar@???: no retry items
Failed addresses:
Deferred addresses:
end of retry processing
LOG: MAIN
Completed
end delivery of 1CRVc5-000Fcm-45
search_tidyup called
search_tidyup called
>>>>>>>>>>>>>>>> Exim pid=61741 terminating with rc=0 >>>>>>>>>>>>>>>>
The behaviour I'm seeing, as shown in the debugging output is that as the
second (and subsequent) addresses are routed to a previous address they
are discarded:
postmaster@??? is a duplicate address: discarded
This happens at line 5095 of deliver.c.
Unless anyone has any better ideas, I'll try to provide Philip a patch to make
the de-duplication at line deliver.c:5095 conditional (probably per
configurable per transport) and hope he might include it in future releases.
If anyone does have a better idea on how to solve this problem, though, I'd be
glad to hear it!
Thanks.
Ollie
--
Ollie Cook Systems Architect, Claranet UK
ollie@??? +44 20 7685 8065
