Jeremy Harris via Exim-users wrote:
> On 30/12/2020 08:04, Evgeniy Berdnikov via Exim-users wrote:
> > On Wed, Dec 30, 2020 at 02:25:19PM +0700, Victor Sudakov via Exim-users wrote:
> > > Is this ktrace informative https://termbin.com/zjsv ?
>
> Yes; thanks.
>
> >
> > 8889 exim CALL socket(PF_INET,0x1<SOCK_STREAM>,IPPROTO_IP)
> > 8889 exim RET socket 5
> > 8889 exim CALL setitimer(0,0x7fffffff30d0,0x7fffffff30b0)
> > 8889 exim STRU itimerval { .interval = {0, 0}, .value = {5, 0} }
> > 8889 exim STRU itimerval { .interval = {0, 0}, .value = {0, 0} }
> > 8889 exim RET setitimer 0
> > 8889 exim CALL setsockopt(0x5,IPPROTO_TCP,TCP_FASTOPEN,0x254270,0x4)
> > 8889 exim RET setsockopt 0
> > 8889 exim CALL sendto(0x5,0x2322d6,0xa,0,0x7fffffff3150,0x10)
> > 8889 exim STRU struct sockaddr { AF_INET, 192.168.153.104:3310 }
> > 8889 exim GIO fd 5 wrote 10 bytes
> > "zINSTREAM\0"
> > 8889 exim RET sendto 10/0xa
> > 8889 exim CALL setitimer(0,0x7fffffff30d0,0x7fffffff30b0)
> > 8889 exim STRU itimerval { .interval = {0, 0}, .value = {0, 0} }
> > 8889 exim STRU itimerval { .interval = {0, 0}, .value = {4, 999970} }
> > 8889 exim RET setitimer 0
> > 8889 exim CALL close(0xffffffff)
> > 8889 exim RET close -1 errno 9 Bad file descriptor
> >
> > As packet is sent, it may be some problem with TCP_FASTOPEN, probably
> > with its handling in hypervisor and/or external firewall.
>
> Kernel, I think. The packet capture showed the SYNs not carrying
> any TFO request, despite that TCP_FASTOPEN setsockopt. Probably the
> FreeBSD implementation has changed since I worked on the Exim
> implementation, in such a way as to break the combination.
>
> In case it helps, my notes from then include:
>
> # FreeBSD: it looks like you have to compile a custom kernel, with
> # 'options TCP_RFC7413' in the config. Also set
> # 'net.inet.tcp.fastopen.server_enable=1' in /etc/sysctl.conf
Well, the defaults in FreeBSD 12.2 (GENERIC kernel) are:
$ sysctl -a | grep fastopen
net.inet.tcp.fastopen.server_enable: 0
net.inet.tcp.fastopen.psk_enable: 0
net.inet.tcp.fastopen.path_disable_time: 900
net.inet.tcp.fastopen.numpsks: 0
net.inet.tcp.fastopen.numkeys: 0
net.inet.tcp.fastopen.maxpsks: 2
net.inet.tcp.fastopen.maxkeys: 2
net.inet.tcp.fastopen.keylen: 16
net.inet.tcp.fastopen.client_enable: 1
net.inet.tcp.fastopen.ccache_buckets: 2048
net.inet.tcp.fastopen.ccache_bucket_limit: 16
net.inet.tcp.fastopen.autokey: 120
In my situation, I set net.inet.tcp.fastopen.client_enable=0 on the
client (exim host) and it cured the problem. I could probably have set
net.inet.tcp.fastopen.server_enable=1 on the server side (clamd host)
and it would cure the problem too?
>
> If there's a sysctl for enabling the client side, try changing
> it. If that affects this, we need to know.
Please see above.
>
>
> I could try to code up a backstop, retrying the connection without
> TFO... sigh. Without some effort it wouldn't be particularly efficient
> in service operation, and I call having to do that "ugly". Better
> to get it working correctly, or deciding before trying that the
> feature is not usable on the platform.
> Slightly less ugly would be a config option "no TFO", either global
> or just on the av_scanner address.
Jeremy, I did not quite understand if the whole problem is a bug in
FreeBSD, or a bug in Exim, or both, but if I can provide any help or
additional info/testing to clear the situation once and for all, I'd be
glad to.
>
>
> Meantime, the Exim debug channels "acl" and "transport" would show
> the sequence from a higher-level view. We might be able to guess
> what that close(-1) was (yes, that's a bug. Not an important one).
>
--
Victor Sudakov, VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet
http://vas.tomsk.ru/
