It may be worth examining the queue_smtp_domains option along with
remote_max_parallel - this can allow you to make best use of a set of
connections to your smarthost without flooding it.
Nigel.
> Jonas Meurer <mailto:jonas@freesources.org>
> 16 November 2015 at 13:43
> Hi,
>
> first, thanks a lot for your answers. It's much appreciated.
>
> Am 2015-11-16 12:26, schrieb Evgeniy Berdnikov:
>> On Mon, Nov 16, 2015 at 10:50:12AM +0000, Jeremy Harris wrote:
>>> On 16/11/15 10:06, Jonas Meurer wrote:
>>> > As you can see, the mails are sent to the smarthost one by one, with
>>> > a new connection for each mail. Is it possible to make Exim4 deliver
>>> > several mails together to the smarthost in one connection?
>>>
>>> Exim normally reuses a connection for multiple deliveries; it takes
>>> specific configuration to make it not do so. This is a Debian
>>> config problem; you need to ask in a Deb-specific place. The Deb
>>> docs should tell you where.
>>
>> Debian Exim package has no specific options for remote_smtp_smarthost
>> transport, limiting number of messages per connection.
>
> The remote_smtp_smarthost transport configuration here is exactly the
> following (macros already expanded):
>
> remote_smtp_smarthost:
> debug_print = "T: remote_smtp_smarthost for $local_part@$domain"
> driver = smtp
> hosts_try_auth = <; ${if exists{/etc/exim4/passwd.client} \
> {\
>
> ${lookup{$host}nwildlsearch{/etc/exim4/passwd.client}{$host_address}}\
> }\
> {} \
> }
>
> No connection_max_messages is set in the whole Exim4 configuration at
> all, so the default of 500 should apply to the remote_smtp_smarthost
> transport, no?
>
>> And I see no proof of single-delivery statement in the initial post,
>> its logs do not show such behaviour. However, some mails are not sent
>> immediately but queued due smtp_accept_queue_per_connection=100, with
>> record in log "no immediate delivery: more than 100 messages received
>> in one connection". Maybe the reason of delays is that some mail
>> bundles
>> are sent to smarthost during scheduled queue runs (once 30 min or
>> more).
>
> Indeed, most messages are queued due to smtp_accept_queue_per_connection
> being set to 100.
>
> The mailserver in question is quiet most of the time, but several times
> a week it receives something like 15k-20k similar newsletter (but not
> identical) messages from a web application. To my understanding, the
> first
> messages are sent immediately until 100 delivery processes are active.
> all remaining messages are queued and sent later by the exim4 queue
> runner.
>
> Is it a good idea to raise smtp_accept_queue_per_connection to a very
> high value or disable it by setting it to 0 in order to make Exim
> deliver all messages immediately?
>
> Apart from that, I don't get, why the queue runner sends queued messages
> one by one each with a new connection to the smarthost. At least, that's
> how I read the logs. How can I distinguish from the logs whether messages
> are sent bundled in one connection or with a new connection each? To me,
> the logs look like the latter is true in my case. No asterisks next to
> H=... and TLS connection details in each delivery log.
>
> Here's another example of Exim sending out some messages from the queue
> to the smarthost:
>
> [...]
> 2015-11-13 14:31:25 1ZxERp-0004ep-DA <=
> melin+N7-IDNL1447420608-C0@??? H=localhost
> (nabu-newsletter) [127.0.0.1] P=esmtp S=60573
> id=1744905331.167881447421485398.JavaMail.root@nabu-newsletter
> 2015-11-13 14:31:25 1ZxERp-0004ep-DA no immediate delivery: more than
> 100 messages received in one connection
> [...]
> 2015-11-13 14:34:20 1ZxEUe-00059s-D7 <=
> melin+N7-IDNL1447420608-C0@??? H=localhost
> (nabu-newsletter) [127.0.0.1] P=esmtp S=112449
> id=805524646.201721447421660403.JavaMail.root@nabu-newsletter
> 2015-11-13 14:34:20 1ZxEUe-00059s-D7 no immediate delivery: more than
> 100 messages received in one connection
> [...]
> 2015-11-13 14:34:51 1ZxEV9-0005JT-OM <=
> melin+N7-IDNL1447420608-C0@??? H=localhost
> (nabu-newsletter) [127.0.0.1] P=esmtp S=95763
> id=1632431573.208141447421691753.JavaMail.root@nabu-newsletter
> 2015-11-13 14:34:51 1ZxEV9-0005JT-OM no immediate delivery: more than
> 100 messages received in one connection
> [...]
> 2015-11-13 14:35:24 1ZxEVf-0005JT-WE <=
> melin+N7-IDNL1447420608-C0@??? H=localhost
> (nabu-newsletter) [127.0.0.1] P=esmtp S=91838
> id=898528339.213911447421723997.JavaMail.root@nabu-newsletter
> 2015-11-13 14:35:24 1ZxEVf-0005JT-WE no immediate delivery: more than
> 100 messages received in one connection
> [...]
> 2015-11-13 15:22:21 1ZxEVf-0005JT-WE Spool file is locked (another
> process is handling this message)
> 2015-11-13 15:22:21 1ZxEVf-0005JT-WE => john.doe@???
> R=smarthost T=remote_smtp_smarthost H=smtp2.eu [195.34.176.148]
> X=TLS1.0:RSA_AES_128_CBC_SHA1:128 DN="OU=GT46541191,OU=See
> www.rapidssl.com/resources/cps (c)14,OU=Domain Control Validated -
> RapidSSL(R),CN=*.inet.de" C="250 2.0.0 tADEMKtb017818 Message accepted
> for delivery"
> 2015-11-13 15:22:21 1ZxEVf-0005JT-WE Completed
> 2015-11-13 15:22:21 1ZxEUe-00059s-D7 => jane.doe@??? R=smarthost
> T=remote_smtp_smarthost H=smtp2.eu [195.34.176.148]
> X=TLS1.0:RSA_AES_128_CBC_SHA1:128 DN="OU=GT46541191,OU=See
> www.rapidssl.com/resources/cps (c)14,OU=Domain Control Validated -
> RapidSSL(R),CN=*.inet.de" C="250 2.0.0 tADEMKhr017821 Message accepted
> for delivery"
> 2015-11-13 15:22:21 1ZxEUe-00059s-D7 Completed
> 2015-11-13 15:22:21 1ZxER6-0004ep-1p Spool file is locked (another
> process is handling this message)
> 2015-11-13 15:22:21 1ZxETe-0004v9-0M Spool file is locked (another
> process is handling this message)
> 2015-11-13 15:22:21 1ZxEV9-0005JT-OM Spool file is locked (another
> process is handling this message)
> 2015-11-13 15:22:21 1ZxETJ-00059s-HF Spool file is locked (another
> process is handling this message)
> 2015-11-13 15:22:21 1ZxEV9-0005JT-OM => another.john.doe@???
> R=smarthost T=remote_smtp_smarthost H=smtp2.eu [195.34.176.148]
> X=TLS1.0:RSA_AES_128_CBC_SHA1:128 DN="OU=GT46541191,OU=See
> www.rapidssl.com/resources/cps (c)14,OU=Domain Control Validated -
> RapidSSL(R),CN=*.inet.de" C="250 2.0.0 tADEMKIG017817 Message accepted
> for delivery"
> 2015-11-13 15:22:21 1ZxEV9-0005JT-OM Completed
> 2015-11-13 15:22:21 1ZxETS-00059s-16 Spool file is locked (another
> process is handling this message)
> 2015-11-13 15:22:21 1ZxETJ-00059s-HF Spool file is locked (another
> process is handling this message)
> 2015-11-13 15:22:21 1ZxERp-0004ep-DA => another.jane.doe@???
> R=smarthost T=remote_smtp_smarthost H=smtp2.eu [195.34.176.148]
> X=TLS1.0:RSA_AES_128_CBC_SHA1:128 DN="OU=GT46541191,OU=See
> www.rapidssl.com/resources/cps (c)14,OU=Domain Control Validated -
> RapidSSL(R),CN=*.inet.de" C="250 2.0.0 tADEMK55017822 Message accepted
> for delivery"
> 2015-11-13 15:22:21 1ZxERp-0004ep-DA Completed
>
> Cheers
> jonas
>
> Evgeniy Berdnikov <mailto:bd4@protva.ru>
> 16 November 2015 at 11:26
>
> Debian Exim package has no specific options for remote_smtp_smarthost
> transport, limiting number of messages per connection.
>
> And I see no proof of single-delivery statement in the initial post,
> its logs do not show such behaviour. However, some mails are not sent
> immediately but queued due smtp_accept_queue_per_connection=100, with
> record in log "no immediate delivery: more than 100 messages received
> in one connection". Maybe the reason of delays is that some mail bundles
> are sent to smarthost during scheduled queue runs (once 30 min or more).
> Jeremy Harris <mailto:jgh@wizmail.org>
> 16 November 2015 at 10:50
>
> Exim normally reuses a connection for multiple deliveries; it takes
> specific configuration to make it not do so. This is a Debian
> config problem; you need to ask in a Deb-specific place. The Deb
> docs should tell you where.
> Jonas Meurer <mailto:jonas@freesources.org>
> 16 November 2015 at 10:06
> Hi Exim users,
>
> for some reason, mail delivery from Exim4 (4.84-8 on an up-to-date Debian
> 8 Jessie system) to the configured smarthost is very slow. Initially I
> used the default Debian Exim4 configuration and set a global smarthost
> for all outgoing mail.
>
> The system in question is a box used to send personalized newsletters,
> i.e. a separate message is created for each recipient. Some newsletters
> have several ten thousand recipients. The mails are created by a web
> application and delivered to Exim4 through the PHP mail() function.
>
> After a newsletter is sent out, I see all outgoing mails in the Exim4
> mail queue. According to the Exim4 mainlog, approximately one mail
> per second is delivered to the smarthost. I honestly don't understand
> why Exim4 delivers only one mail per second, e.g. why mails are sent
> out that slowly.
>
> Here is an example from the mainlog with Exim4 sending some of the
> mails from the queue:
>
> 2015-11-13 14:13:03 1ZxEA2-0002TF-VT <=
> melin+N7-IDNL1447420377-C0@??? H=localhost
> (example-newsletter) [127.0.0.1] P=esmtp S=64814
> id=1038582402.1681447420382973.JavaMail.root@example-newsletter
> 2015-11-13 14:13:03 1ZxEA3-0002TF-1F <=
> melin+N7-IDNL1447420377-C0@??? H=localhost
> (example-newsletter) [127.0.0.1] P=esmtp S=16799
> id=2069093441.1691447420383037.JavaMail.root@example-newsletter
> [...]
> 2015-11-13 14:13:04 1ZxEA2-0002TF-VT => user.name@???
> R=smarthost T=remote_smtp_smarthost H=smtp2.eu [195.34.176.148]
> X=TLS1.0:RSA_AES_128_CBC_SHA1:128 DN="OU=GT46541191,OU=See
> www.rapidssl.com/resources/cps (c)14,OU=Domain Control Validated -
> RapidSSL(R),CN=*.inet.de" C="250 2.0.0 tADDD3NM028115 Message accepted
> for delivery"
> 2015-11-13 14:13:04 1ZxEA2-0002TF-VT Completed
> [...]
> 2015-11-13 14:17:13 1ZxEE5-0002YF-D7 <=
> melin+N7-IDNL1447420608-C0@??? H=localhost
> (example-newsletter) [127.0.0.1] P=esmtp S=60583
> id=757069355.4901447420633404.JavaMail.root@example-newsletter
> 2015-11-13 14:17:13 1ZxEE5-0002YF-D7 no immediate delivery: more than
> 100 messages received in one connection
> [...]
> 2015-11-13 14:44:47 1ZxEE5-0002YF-D7 => user.name@???
> R=smarthost T=remote_smtp_smarthost H=smtp2.eu [195.34.176.148]
> X=TLS1.0:RSA_AES_128_CBC_SHA1:128 DN="OU=GT46541191,OU=See
> www.rapidssl.com/resources/cps (c)14,OU=Domain Control Validated -
> RapidSSL(R),CN=*.inet.de" C="250 2.0.0 tADDik9H020819 Message accepted
> for delivery"
> 2015-11-13 14:44:47 1ZxEE5-0002YF-D7 Completed
>
> As you can see, the mails are sent to the smarthost one by one, with
> a new connection for each mail. Is it possible to make Exim4 deliver
> several mails together to the smarthost in one connection?
>
> The outgoing smarthost is definitely not the problem, it takes mail
> from other satellite mail servers much faster.
>
> I already tried to speed up mail delivery using the following settings,
> which didn't help at all:
>
> smtp_accept_queue_per_connection = 100
> remote_max_parallel = 30
> queue_run_max = 10
>
> Any suggestions?
>
> Cheers
> jonas
>
--
[ Nigel Metheringham ------------------------------ nigel@??? ]
[ Ellipsis Intangible Technologies ]
