Re: [exim] RCPT delays and PIPELINING

Philip Hazel wrote:
> I thought I'd post this so that it gets into the archive. Sites that use
> the "delay" modifier in the RCPT ACL (either in conjunction with
> "ratelimit" or otherwise) might run into problems with PIPELINING.
>
> The PIPELINING extension to SMTP is supposed to make better use of the
> network. Using PIPELINING, a client can send MAIL, RCPT, RCPT, ..., DATA
> all at once, without waiting for a response in between. The client then
> waits for the responses to come back, in order. The server does not need
> to flush its output buffer until it has run out of input from the
> client. The idea is that in many cases, all the commands from MAIL to
> DATA will fit into a single packet, and likewise all the responses will
> go back in a single packet. Without PIPELINING, there is a packet
> exchange for each command.
>
> Exim, as a server, advertises PIPELINING by default, and as a client, it
> uses PIPELINING if the server advertises it. You can suppress the
> advertisement (globally, or to specific hosts) by setting
> pipelining_advertise_hosts.
>
> A problem arises if the server starts applying delays when it responds
> to RCPT commands, because the client won't see any response at all until
> either the server's output buffer fills up, or DATA has been processed.
> For example, if the client sends MAIL, RCPT, RCPT, RCPT, DATA all in one
> packet, and the server applies a 1-minute delay to each RCPT, the client
> will see a 3-minute delay before any response at all is received (it
> will be waiting for a response to MAIL). It doesn't take too many small
> delays to add up to 5 minutes - which is a typical client timeout time.
>
> PIPELINING was invented to speed things up; delays were invented to slow
> things down. They don't play well together. :-(
>
> A feature of the next release of Exim (already in the snapshots) is
>
> control = no_pipelining
>
> This makes it possible to disable pipelining for certain clients based
> on any criteria that an ACL supports, not just the host identity (which
> is all you have with pipelining_advertise_hosts). However, you have to
> disable before the response to EHLO is sent in order for it to work.
>
> Bottom line: if you are using delays with RCPT commands, consider the
> PIPELINING consequences.
>
> A thought: as I wrote this, it occurred to me that maybe there is a way
> that Exim can take note of having obeyed "delay", and if so, force out
> its output buffer after it has written the next response, instead of
> waiting until it has run out of input. If possible, that would solve
> this issue automatically. Any views on this idea?
>

Yes.

From the time we discovered that Exim could do so, we have had advertisement of
pipelining OFF, enforce sync ON, and drop sync requirement on successful auth
for submission. Works well, and catchs 'some few' garbage servers out. Bothers
legitimate servers not at all.

So - never had the conflict, even when using delays.

The benefits of pipelining, OTOH, can no doubt be demonstrated.

BUT - with all the other things going on, I suspect that for some very high
percentage of real-world traffic, either one side or the other will not
support/make use of or gain much from pipelining *anyway*. Rationale is the
prevalence of single-recipient traffic, where there is little to gain.

Grant - it may well be a Godsend for *really* heavy traffic 'routes' - say
between and among gmail, MSN, Yahoo user communities. Or perhaps, in 'Exim land'
among busy university servers.

So I don't 'recommend' disabling pipelining. In the default, or otherwise.

And I do see value in 'more tools'. Generally.

But there is probably more meat on other bones than to add finer granularity w/r
mixing delays and pipelining.

I suspect most who use delays do as we do - NOT work pipelining at all - and
would not change that even if Exim supported a 'mix'.

JM2CW

Bill