Re: [exim] GMail and TCP Fast Open

On 03/12/2021 21:36, Evgeniy Berdnikov via Exim-users wrote:
> Hello.
>
> On Fri, Dec 03, 2021 at 12:00:33PM +0000, Andrew C Aitchison via
Exim-users wrote:
>>> We currently have about 1,000 emails queued for google accounts
>>> and these are failing with a timeout, eg from exim:
>>>
>>> 2021-12-02 11:48:05 1msize-005feQ-DI
>>> H=gmail-smtp-in.l.google.com [142.250.13.26] TLS error on
>>> connection (send): The TLS connection was non-properly
>>> terminated.
>>>
>>> 2021-12-02 11:48:05 1msize-005feQ-DI
>>> H=gmail-smtp-in.l.google.com [142.250.13.26]: SMTP timeout after
>>> sending data block (525043 bytes written): Connection timed out
>
> Looks as broken Path MTU Discovery.
>
>>> Based on past experience, eventually, the emails may will
>>> through. This is also intermittent, as sending to my own gmail
>>> address tends to get through eventually, though they do still
>>> time out once or twice before eventually getting through.
>
> If some mails could be passed, this is a hint that path is changing
> over time, so effect is not stable. With MTR utility path can be
> monitored in real time (like traceroute, but much better).
>
>>> This report is rather vague, but we do have further examples,
>>> logs, pcaps.
>
> Use pcaps, study the moments when connections are timed out.
>
>>> After trying a few things, disabling TCP Fast Open seems to have
worked...
>
> TFO affects connection open stage, but this problem is definitely due
> to timeout in the middle of data transfer.
>
> Many internet sites (may be millions) communicate with Google, so
> this problem is very likely local for one site and its internet
> connection.
>



Thanks for the reply...

We have a direct interconnect to Google, so there is only a fibre
between our router and Google's router, that part of path is stable and
identical - though there are a number of hops withing Google's AS that
could be changing.

ping/mtr - no packet loss

Tried reducing MTU (can't pmtu discover to gmail as they block too much
ICMP) - no difference

Checked IPv4 vs IPv6 - no difference


Andrew.