Hi Mike,
When I said I was hoping exim would defer the message, I meant that the
message would remain in the exim queue and be retried during the normal
retry intervals. Isn't that what would happen? Your details about the
software being able to handle the deferral indicates it wouldn't, and it
would be up to the software injecting the message to handle the retries.
Why wouldn't exim just keep the message in the queue and retry it on the
normal retry schedule if a verification fails/defers?
I had the same suspicions as you about Exim not doing the verify before
it sends the message to the smarthost. I did the test using your command
with the arguments you mentioned, and I'm just confused now. During
startup, the output is normal, and it stops at "Listening...". That seems
normal so far. After that, I ran the command to inject a message into the
queue ("echo test | mail -s test someone@???"),
and nothing showed up in the debug output. Just "Listening..." still.
However, checking /var/log/exim4/mainlog I can see the message gets
delivered successfully (to the relay at least). That makes no sense. How
did exim deliver the message when the debug output never showed a thing?
Thanks,
-- Jason
On Wed, Aug 3, 2016 at 3:16 PM, Mike Brudenell <mike.brudenell@???>
wrote:
> Hi, Jason -
>
> I'm at home and still pondering… :-(
>
> *Important: *If you're injecting your messages directly rather than via
> SMTP you'll have to check how your software reacts if it has a recipient or
> message deferred. Well-behaved software will note the defer and hold the
> message to try again later (do you *really* want that to happen?);
> however other software may not expect to ever get a defer and will just
> abandon the message/copy to that recipient.
>
>
> There's a private option to the *dnslookup* router that might also be of
> use: *fail_defer_domains*. This tells the router to defer rather than
> decline. Check *17 The dnslookup router
> <http://www.exim.org/exim-html-current/doc/html/spec_html/ch-the_dnslookup_router.html>*
> for more information. Because you want it to apply to all domains you'd
> want to assign it the value ***
>
> However something I'm wondering is whether your *verify_address* router
> is actually called at all when attempting a real delivery! I have a vague
> feeling that it won't be unless you're doing a *verify = recipient* in an
> ACL somewhere. (Can anyone else confirm this?)
>
> I'm suspecting that without a *verify = recipient* then your routers
> aren't getting triggered in verification mode at all, but instead going
> straight into the actual routing. (Of course when you run Exim from the
> command line and specify *-bv* you're forcibly running it in verify mode
> for the address you give.)
>
> You could check what's happening by running Exim in debug mode as a daemon
> then sending it your test message and checking through the debugging output
> to check the exact sequence of routers that's called, what tests each ACL
> and router are doing and their outcomes. It's an incredibly powerful
> technique:
>
> exim -v -d+all -bd
>
>
> Make sure you don't already have Exim running when you do this else this
> test daemon won't be able to bind to the ports.
>
> You could cut down the amount of logging data by using a different option
> to *-d*; however I tend to like seeing all the gory details to make sure
> I know exactly what's being checked/happening and in what order.
>
>
> I have this (possibly incorrect) belief that Exim is so flexible that
> there *will* be a way to do what you want. However it may not be using
> the route we've started down, but require some sideways thinking. Anyone
> else want to chip in?
>
> Cheers,
> Mike B-)
>
> On 3 August 2016 at 19:13, Jason <silo82@???> wrote:
>
>> I did try to put *no_more* on the *verify_address* router, but it made no
>> difference when sending an actual message. It still failed the *-bv* test,
>> but the message was still passed to the smarthost router and sent out to
>> the relay. The way it's described, using *no_more* is exactly what I'm
>> looking for...check DNS, if it fails for whatever reason, fail/defer the
>> message and don't pass it to the next router. I'm not sure why it's not
>> doing that, though.
>>
>> I haven't looked into the *acl_not_smtp*, but will do that. I'm guessing
>> that's used for messages that get directly injected into the queue using
>> mail() functions/programs. That could work, but I'd rather defer the
>> message rather than permfail it, simply because it's possible DNS lookups
>> may fail due to remote server issues, network issues, etc., and the next
>> time the messages gets retried DNS may be working again.
>>
>> Good tip for *ignore_target_hosts*!
>>
>>
>>
>> On Wed, Aug 3, 2016 at 2:03 PM, Mike Brudenell <mike.brudenell@???
>> > wrote:
>>
>>> Hi, Jason -
>>>
>>> If a *dnslookup* router decides it can't handle an address it declines
>>> (as described in the *Specification*) and control passes to the next
>>> router. This could be useful for, say, a host with limited Internet
>>> connectivity: its DNS lookups fail but it can then fall through to a
>>> *manualroute* router and send the message off to a host with better
>>> connectivity.
>>>
>>> If you want a *dnslookup* router to handle "all remaining non-local
>>> domains" and not decline/fall through to the next router you must set the
>>> *no_more* option on it. (See *17.2. Declining addresses by dnslookup
>>> <http://www.exim.org/exim-html-current/doc/html/spec_html/ch-the_dnslookup_router.html#SECTdnslookupdecline>*)
>>> This in effect turns the router's *decline* into a *fail*. (See *3.10 Running
>>> an individual router
>>> <http://www.exim.org/exim-html-current/doc/html/spec_html/ch-how_exim_receives_and_delivers_mail.html#SECTrunindrou>*
>>> )
>>>
>>>
>>> Would you be better off doing your cleanup earlier, in the ACLs, rather
>>> than trying to force it into the routers? If you do then you can refuse to
>>> accept that recipient address when the message is first trying to enter
>>> your server, as against accepting it and then trying to get rid of it.
>>>
>>> For example, perhaps something like this *untested* entry within your
>>> *acl_smtp_rcpt* ACL?
>>>
>>> require verify = recipient
>>>
>>>
>>> 1. The *require* says that its conditions *must* be met otherwise it
>>> will reject the recipient address. (I guess you could do "defer ! verify =
>>> recipient" instead if you really wanted to, with the risks Lena points out.)
>>>
>>> 2. The *verify = recipient* says that the recipient address must
>>> verify by your routers in order to succeed. And you've already got working
>>> routers to do that, I think: your *dnslookup* router is used only
>>> during verification and can check the MX hosts look sensible, but during
>>> delivery you instead throw the message at your smarthost using your
>>> *manualroute* router for actual delivery.
>>>
>>> Note that by default verification only checks out the domain part of
>>> the address; you'd have to add */callout* if you wanted to to a
>>> forward callout to the far mail server and get it to confirm the localpart
>>> of the address is valid too.
>>>
>>> *Hmm…*
>>>
>>> I've just seen your update saying that the messages *aren't* arriving
>>> over SMTP so that the acl_smtp_rcpt ACL isn't any use to you… True, so is
>>> there any way of you doing the check in an *acl_not_smtp* ACL instead?
>>> (Although do note that the *Specification* says, "Any kind of rejection
>>> is treated as permanent, because there is no way of sending a temporary
>>> error for these kinds of message.")
>>>
>>> I need to head home shortly, but will keep pondering; no doubt others
>>> might get back to you in the meantime.
>>>
>>>
>>> *By the way…*
>>>
>>> If you're cleaning up your outbound mail make sure you read up and use
>>> *ignore_target_hosts* as well in your *dnslookup* router. Some rogue
>>> domains point their MX/A/AAAA records to loopback (eg, 0 or 127.0.0.1) or
>>> private IP addresses, so they appear to have entries but in fact they're of
>>> no use. So you want to filter those out from any entries your
>>> *dnslookup* router retrieves to check.
>>>
>>> For example:
>>>
>>> *…Main configuration…*
>>> hostlist private_ip_addrs = <; 10.0.0.0/8 ; 172.16.0.0/12 ;
>>> 192.168.0.0/16
>>> hostlist loopback_ip_addrs = <; 0.0.0.0 ; 127.0.0.0/8 ; ::1
>>>
>>> *…Routers…*
>>> verify_host:
>>>
>>> driver = dnslookup
>>>
>>> ignore_target_hosts = +loopback_ip_addrs : +private_ip_addrs
>>> …
>>>
>>>
>>> Cheers,
>>> Mike B-)
>>>
>>> On 3 August 2016 at 18:29, Jason <silo82@???> wrote:
>>>
>>>> Thanks, Mike. I'll check out the lookup. So from the way you
>>>> described it, when sending mail (not testing with -bv or -bt), it hits the
>>>> "verify_address" router, does the check, and would fail, but it still
>>>> continues to the next router ("smarthost" in my case) which delivers the
>>>> message? Shouldn't the verify_address router failure halt/defer the
>>>> message and not send it onto the next router? If not, is that possible in
>>>> the router configuration (without a lookup)? What purpose does a
>>>> verify_only router serve if it doesn't defer/fail a message from being
>>>> processed further?
>>>>
>>>> The reason we want to do the check before forwarding to the relay is
>>>> because our relay is Amazon SES, and if we relay mail that's destined to
>>>> NXDOMAINs, for example, then it will be a bounce and you get penalized for
>>>> submitting mail that bounces (we're trying to cleanup email addresses that
>>>> are in our database that are entered incorrectly, but for now, they're
>>>> there). So we're trying to deal with the "bad" mail before it even gets to
>>>> SES.
>>>>
>>>> Thanks,
>>>> -- Jason
>>>>
>>>> On Wed, Aug 3, 2016 at 1:04 PM, Mike Brudenell <
>>>> mike.brudenell@???> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Exim is doing exactly what your routers are currently written to tell
>>>>> it do…
>>>>>
>>>>> Your *verify_address* router is set (by the *verify_only*) to be used
>>>>> only when verifying an address: for example, as you're doing when you use
>>>>> the "-bv" command line option. If the DNS lookup says that the domain
>>>>> doesn't exist then the router declines to handle the message and it falls
>>>>> through to the next router.
>>>>>
>>>>> The next router is *smarthost* which is set not to be used when
>>>>> verifying (because you have given it *no_verify*). So when you're
>>>>> verifying there's no later router that handles the address being verified
>>>>> so you get the "Unrouteable address" result.
>>>>>
>>>>> *However* when you're doing it for real then when *verify_address* finds
>>>>> a domain that doesn't exist in the DNS and declines, the *smarthost* router
>>>>> *does* run and do exactly what you tell it: send the message over to
>>>>> your smarthost. (I've not used *host_find_failed* but suspect it
>>>>> applies if Exim can't find IP addresses for the smarthost(s) you've defined
>>>>> in *route_list*, not for the recipient address' domain.)
>>>>>
>>>>> As it says in at the start of the *Specification* when describing a
>>>>> *manualroute* router (emphasis by underlining mine)…
>>>>>
>>>>> The *manualroute* router is so-called because it provides a way of
>>>>> manually routing an address according to its domain. It is mainly used when
>>>>> you want to route addresses to remote hosts according to your own rules, *bypassing
>>>>> the normal DNS routing that looks up MX records*.
>>>>>
>>>>>
>>>>> I have a feeling that you trying to check in your *smarthost* router whether
>>>>> the MX records for the recipient address' domains are valid might be
>>>>> unusual; it's easy to argue that should be the job of the smarthost rather
>>>>> than yourself!
>>>>>
>>>>> But if you're determined to do it then presumably you could use a
>>>>>
>>>>> *${lookup dnsdb{mx=$domain}{$value}fail}*
>>>>>
>>>>> (Someone else might be able to suggest an easier way of doing the
>>>>> check, but this could be one way.)
>>>>>
>>>>> Read the chapter *File and database lookups
>>>>> <http://www.exim.org/exim-html-current/doc/html/spec_html/ch-file_and_database_lookups.html>*
>>>>> in the *Specification* for more on theis and other types of lookup.
>>>>>
>>>>> Cheers,
>>>>> Mike B-)
>>>>>
>>>>> On 3 August 2016 at 14:10, Jason <silo82@???> wrote:
>>>>>
>>>>>> Hi,
>>>>>> Is it possible for Exim to verify outgoing mail when using a
>>>>>> smarthost?
>>>>>> Specifically, I'm looking to have Exim verify (using dnslookup) that
>>>>>> the
>>>>>> domain exists for the recipient. If the domain returns NXDOMAIN (or
>>>>>> some
>>>>>> other DNS error), defer it, if DNS exists, continue with the
>>>>>> routing. I
>>>>>> tried the following:
>>>>>>
>>>>>> # Verify the recipient domain exists in DNS before sending to the
>>>>>> relay.
>>>>>> verify_address:
>>>>>> debug_print = "R: verify_address for $domain"
>>>>>> driver = dnslookup
>>>>>> domains = ! +local_domains
>>>>>> verify_only
>>>>>> no_more
>>>>>>
>>>>>>
>>>>>> smarthost:
>>>>>> debug_print = "R: smarthost for $local_part@$domain"
>>>>>> driver = manualroute
>>>>>> domains = ! +local_domains
>>>>>> transport = remote_smtp_smarthost
>>>>>> route_list = * DCsmarthost byname
>>>>>> host_find_failed = defer
>>>>>> same_domain_copy_routing = yes
>>>>>> no_verify
>>>>>> no_more
>>>>>>
>>>>>> And it seems to work during testing:
>>>>>>
>>>>>> # exim -bv someone@???
>>>>>> someone@??? failed to verify: Unrouteable
>>>>>> address
>>>>>> # exim -bv someone@???
>>>>>> someone@??? verified
>>>>>>
>>>>>>
>>>>>> However, when sending a test message to an NXDOMAIN, exim happily
>>>>>> forwards
>>>>>> it to the smarthost:
>>>>>>
>>>>>> 2016-08-03 08:07:28 1bUvtQ-0007bm-Vh <= root@somehost U=root P=local
>>>>>> S=486
>>>>>> T="test"
>>>>>> 2016-08-03 08:07:29 1bUvtQ-0007bm-Vh =>
>>>>>> someone@??? R=smarthost
>>>>>> T=remote_smtp_smarthost
>>>>>> S=505 H=ses-smtp-prod-335357831.us-east-1.elb.amazonaws.com
>>>>>> [50.19.94.229]
>>>>>> X=TLS1.2:DHE_RSA_AES_128_CBC_SHA1:128
>>>>>> DN="C=US,ST=Washington,L=Seattle,O=Amazon.com\, Inc.,CN=
>>>>>> email-smtp.us-east-1.amazonaws.com"
>>>>>> 2016-08-03 08:07:29 1bUvtQ-0007bm-Vh Completed
>>>>>>
>>>>>> Is it possible for Exim to do outbound verification while using a
>>>>>> smarthost? If so, what am I missing?
>>>>>>
>>>>>> Thanks
>>>>>> --
>>>>>> ## List details at https://lists.exim.org/mailman/listinfo/exim-users
>>>>>> ## Exim details at http://www.exim.org/
>>>>>> ## Please use the Wiki with this list - http://wiki.exim.org/
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Systems Administrator & Change Manager
>>>>> IT Services, University of York, Heslington, York YO10 5DD, UK
>>>>> Tel: +44-(0)1904-323811
>>>>>
>>>>> Web: www.york.ac.uk/it-services
>>>>> Disclaimer: www.york.ac.uk/docs/disclaimer/email.htm
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Systems Administrator & Change Manager
>>> IT Services, University of York, Heslington, York YO10 5DD, UK
>>> Tel: +44-(0)1904-323811
>>>
>>> Web: www.york.ac.uk/it-services
>>> Disclaimer: www.york.ac.uk/docs/disclaimer/email.htm
>>>
>>
>>
>
>
> --
> Systems Administrator & Change Manager
> IT Services, University of York, Heslington, York YO10 5DD, UK
> Tel: +44-(0)1904-323811
>
> Web: www.york.ac.uk/it-services
> Disclaimer: www.york.ac.uk/docs/disclaimer/email.htm
>
