I think I can see what's happening now, but not how to fix it…
1. I have 5 test messages in the queue going to my mailbox.
2. I delete the ratelimit and retry hints databases so I'm starting from
a clean base.
3. I do an "exim -d+all -qq"
4. The first routing run of the -qq option activates and accepts the
first couple of messages, increasing the computed ratelimit as it does so.
This reaches the limit I've set, so the remaining messages are then
deferred. So far so good.
5. The second routing run of the -qq option then activates to perform
deliveries. This routes each message *again*, uses the ratelimiting
router and in doing so sees the high computed ratelimit so now *all* of
the messages now get deferred.
There might also be an (as yet unconfirmed) problem with my setup in that
if the first pass didn't reach the rate limit then the second pass might
when it re-routes the messages: effectively counting each message twice
towards the computed rate (once in the first pass, once in the second). If
so, then some or all of the messages in the second pass might be deferred
when really they should be delivered.
I think I need to stop the second pass of the -qq queue runner from using
this router again? Is there a way of doing that? Or have I got the
fundamental design of router and ACL wrong?
Cheers,
Mike B-)
On 28 February 2018 at 17:06, Mike Brudenell <mike.brudenell@???>
wrote:
> Bug/RFE: I'll have a think about whether I can phrase something.
>
> In the meantime, I've been playing around with the outline Jeremy
> suggested on the Wiki and thought I had it working. Indeed my testing with
> an artificially low rate of 2 messages per 1 minute showed:
>
> - submitting several messages to Exim let the first few through, then
> kept the rest in the queue marked "deferred"
> - running "exim -q" periodically allowed more to be delivered whenever
> the calculate rate dropped sufficiently
>
> So far so good.
> *But…* then I let my test server do its own thing and messages didn't go
> out from the queue! :-(
>
> After a lot of digging I discovered that the problem lies with my queue
> runners being launched using Exim's "-qq2m" option rather than "-q2m",
> which I do for efficiency purposes as some hosts I deliver to limit the
> number of concurrent connections.
>
> Peering closely at the debugging output from "exim -qq -d+all" it appears
> that the double queue run the -qq option triggers is doing something odd:
> possibly updating the ratelimit database twice, causing my artificially low
> rate limit to be reached before a single message is actually delivered. As
> the Specification says,
>
>
> In the first stage, the queue is scanned as if the queue_smtp_domains
> option matched every domain. Addresses are routed, local deliveries happen,
> but no remote transports are run.
>
> The hints database that remembers which messages are waiting for specific
> hosts is updated, as if delivery to those hosts had been deferred. After
> this is complete, a second, normal queue scan happens, with routing and
> delivery taking place as normal.
>
>
> I have no_verify set on my router, but of course this only stops it
> activating during address verification. With the -qq option Exim is
> actually doing two full routing runs.
>
> *Question:* Is there some option I've not spotted that says something
> along the lines of "update the ratelimit database for the first run of the
> -qq option but not the second"?
>
> In case it helps, here is my ACL:
>
> acl_ratelimit_outbound:
> accept message = :defer: Sending messages to $acl_arg1 too fast:
> deferred \
> [$sender_rate/$sender_rate_period (max
> $sender_rate_limit/$sender_rate_period)]
> ratelimit = $acl_arg2 / readonly / outbound:${lc:$acl_arg1}
>
> warn ratelimit = $acl_arg2 / strict / outbound:${lc:$acl_arg1}
>
> accept message = $acl_arg1
>
>
> and my router:
>
> ratelimit_outbound:
> driver = redirect
> no_verify
> allow_defer
> condition = ${if exists {CFG_D/ratelimit-table}}
> data = ${lookup {$local_part@$domain} lsearch*@
> {CFG_D/ratelimit-table} \
> { ${acl {acl_ratelimit_outbound} {$local_part@$domain}
> {$value}} } \
> fail}
>
>
> Cheers,
> Mike B-)
>
> On 28 February 2018 at 10:34, Jeremy Harris via Exim-users <
> exim-users@???> wrote:
>
>> On 28/02/18 10:17, Mike Brudenell via Exim-users wrote:
>> > So despite being a profound advocate of Exim, it is a little
>> embarrassing
>> > to tell people who ask me about outbound rate limiting that it's so
>> > difficult in it
>> Perhaps you could come up with a clean set of requirements?
>> In a bug/RFE, so that others could chip in, and the need
>> wouldn't be forgotten?
>> --
>> Cheers,
>> Jeremy
>>
>> --
>> ## List details at https://lists.exim.org/mailman/listinfo/exim-users
>> ## Exim details at http://www.exim.org/
>> ## Please use the Wiki with this list - http://wiki.exim.org/
>>
>
> --
> Systems Administrator & Change Manager
> IT Services, University of York, Heslington, York YO10 5DD, UK
> Tel: +44-(0)1904-323811 <01904%20323811>
>
> Web: www.york.ac.uk/it-services
> Disclaimer: www.york.ac.uk/docs/disclaimer/email.htm
>
--
Systems Administrator & Change Manager
IT Services, University of York, Heslington, York YO10 5DD, UK
Tel: +44-(0)1904-323811
Web:
www.york.ac.uk/it-services
Disclaimer:
www.york.ac.uk/docs/disclaimer/email.htm
