> Matthew Byng-Maddick wrote:
>
> On Thu, Apr 22, 2004 at 10:54:12AM -0500, Edgar Lovecraft wrote:
> > Matthew Byng-Maddick wrote:
> > > While looking at PIPELINING support within SAUCE last night, the
> > > following issues were highlit by Ian Jackson. The RFC2920 (STD 60)
> > > spec for SMTP command pipelining includes no precise definition of a
> > > "command group",
> >
> > How so? the RFC does not define 'what exactly is in a group', it does
> > however define quite clearly what commands ARE ONLY last command, no
> > matter what is in the group, these commands ARE ALWAYS defined as last.
>
> Imagine that I send (in one TCP command) all the commands in a group. It
> doesn't *actually* say I need to wait for an answer.
Section 3.1 Paragraph 1 and 4
<START OF RFC SNIPET>
P.1
The EHLO, DATA, VRFY, EXPN, TURN,QUIT, and NOOP commands can only appear
as the last command in a group since their success or failure produces a
change of state which the client SMTP must accommodate.
P.4
Client SMTP implementations that employ pipelining MUST check ALL
statuses associated with each command in a group.
</START OF RFC SNIPET>
You must wait. Also, if you are suggesting that your 'group' also
contains the actual DATA of the message then your client is broken.
Section 3.1 Paragraph 4
<START OF RFC SNIPET>
If the DATA command was properly rejected the client SMTP can just
issue RSET, but if the DATA command was accepted the client SMTP
should send a single dot.
</START OF RFC SNIPET>
If your client does not wait to check all status codes as it MUST do,
then how does your client know to send a single dot or not?
> > Lets look at Section 3.1
> > <START OF RFC SNIPET>
> > 3.1. Client use of pipelining
> > Once the client SMTP has confirmed that support exists for the
> > pipelining extension, the client SMTP may then elect to transmit
> > groups of SMTP commands in batches without waiting for a response to
> > each individual command. In particular, the commands RSET, MAIL
> > FROM, SEND FROM, SOML FROM, SAML FROM, and RCPT TO can all appear
> > anywhere in a pipelined command group. The EHLO, DATA, VRFY, EXPN,
> > TURN, QUIT, and NOOP commands can only appear as the last command in
> > a group since their success or failure produces a change of state
> > which the client SMTP must accommodate. (NOOP is included in this
> > group so it can be used as a synchronization point.)
> >
> > Additional commands added by other SMTP extensions may only appear
> > as the last command in a group unless otherwise specified by the
> > extensions that define the commands.
> > </START OF RFC SNIPET>
> > That seems pretty clear to me, as there are some commands that MUST and
> > CAN ONLY be the last of a group (a group can be one command by the
> > way).
>
> CAN ONLY is not part of RFC2119, and the word MUST does not appear in
> that text. It seems pretty clear until you actually look at the language,
> which doesn't prohibit the scenario I've outlined above. The RFC is
> confused, and full of layer violations (expecting the Layer 4 stuff to
> know about what's going on at Layer 3).
Yes it does.
Section 3.1 Paragraph 1
<START OF RFC SNIPET>
The EHLO, DATA, VRFY, EXPN, TURN, QUIT, and NOOP commands can only
^^^^^^^^
appear as the last command in a group since their success or failure
produces a change of state which the client SMTP must accommodate.
(NOOP is included in this group so it can be used as a synchronization
point.)
</START OF RFC SNIPET>
Unless I do not know my English, "can only" means that those commands
are ONLY allowed to be used as a LAST command in a command group. As I
said before, there is no restriction on how many commands my be in a
command group, so therefore, an EHLO or DATA command can be a command
group in and of themselves, so could a MAIL FROM, or RCPT TO, in either
case, when a client ends a command group with any command it MUST check
ALL statuses associated with each command in that group.
> > Lets continue with Section 3.1...
> > <START OF RFC SNIPET>
> > The actual transfer of message content is explicitly allowed to be
> > the first "command" in a group. That is, a RSET/MAIL FROM sequence
> > used to initiate a new message transaction can be placed in the same
> > group as the final transfer of the headers and body of the previous
> > message.
> >
> > Client SMTP implementations that employ pipelining MUST check ALL
> > statuses associated with each command in a group. For example, if
> > none of the RCPT TO recipient addresses were accepted the client
> > must then check the response to the DATA command -- the client
> > cannot assume that the DATA command will be rejected just because
> > none of the RCPT TO commands worked. If the DATA command was
> > properly rejected the client SMTP can just issue RSET, but if the
> > DATA command was accepted the client SMTP should send a single dot.
> >
> > Command statuses MUST be coordinated with responses by counting each
> > separate response and correlating that count with the number of
> > commands known to have been issued. Multiline responses MUST be
> > supported. Matching on the basis of either the error code value or
> > associated text is expressly forbidden.
> > </START OF RFC SNIPET>
> > Here again, the 'what exactly is in a group' is not defined, but what
> > starts or ends the group is defined, as are the rules that the client
> > must follow in regard to response codes.
>
> Yes, but at no point does it explicitly say that a command group
> terminator MUST wait for a response. Please point out the part where it
> says this. It only says that it must be the end of a TCP send operation.
Section 3.1 Paragraph 1 and 4
<START OF RFC SNIPET>
P.1
The EHLO, DATA, VRFY, EXPN, TURN,QUIT, and NOOP commands can only appear
as the last command in a group since their success or failure produces a
change of state which the client SMTP must accommodate.
P.4
Client SMTP implementations that employ pipelining MUST check ALL
statuses associated with each command in a group.
</START OF RFC SNIPET>
If your client does not wait, then how can it check 'ALL statuses' as it
MUST do?
<INSERT RESPONSE FROM TONY>
> It's implicit in the state-transition business, e.g. the way the client
> has to handle DATA, and the mention that NOOP is a synchronization point.
> Yes it could be clearer.
</INSERT RESPONSE FROM TONY FINCH>
I would only agree that it could be clearer if I did not have the
understanding that RFC compliance is more than just understanding any
ONE RFC. It all seems very clear to me.
> > > and in particular, although it's implied that the "last command
> > > of a command group" is one that should wait for replies, the server:
> > Not just implied, the client MUST wait for all replies.
>
> Quote, please.
Section 3.1 Paragraph 1 and 4
<START OF RFC SNIPET>
P.1
The EHLO, DATA, VRFY, EXPN, TURN,QUIT, and NOOP commands can only appear
as the last command in a group since their success or failure produces a
change of state which the client SMTP must accommodate.
P.4
Client SMTP implementations that employ pipelining MUST check ALL
statuses associated with each command in a group.
</START OF RFC SNIPET>
> > > | (9) MUST NOT flush or otherwise lose the contents of the TCP input
> > > | buffer under any circumstances whatsoever.
> > That is to ensure that the server has seen the entire 'command group'
> > sent by the client, it has nothing to with the client not waiting for
> > ALL responses to the 'command group' which the client MUST do. There
> > are also
>
> Quote, please.
Section 3.1 Paragraph 1 and 4
<START OF RFC SNIPET>
P.1
The EHLO, DATA, VRFY, EXPN, TURN,QUIT, and NOOP commands can only appear
as the last command in a group since their success or failure produces a
change of state which the client SMTP must accommodate.
P.4
Client SMTP implementations that employ pipelining MUST check ALL
statuses associated with each command in a group.
</START OF RFC SNIPET>
> It may be "to ensure...." however, if you think through what is
> happening, because there's no restriction on the command group actually
> having to wait for a reply (despite your assertions of a MUST, it doesn't
> exist), it also means that you can't legally (as a server) throw away
> stuff that exim currently considers to be a pipelining violation.
Please explain what things exim currently throws away and considers to
be pipelining violations that you do not agree with.
<START OF SMTP_IN.C FROM EXIM 4.32 SOURCE>
/* Codes for identifying commands. We order them so that those that come
first are those for which synchronization is always required. Checking this
can help block some spam. */
enum {
/* These commands are required to be synchronized, i.e. to be the last in
a block of commands when pipelining. */
HELO_CMD, EHLO_CMD, DATA_CMD, /* These are listed in the pipelining */
VRFY_CMD, EXPN_CMD, NOOP_CMD, /* RFC as requiring synchronization */
ETRN_CMD, /* This by analogy with TURN from the RFC */
STARTTLS_CMD, /* Required by the STARTTLS RFC */
</START OF SMTP_IN.C FROM EXIM 4.32 SOURCE>
These all look good to me.
<START OF SMTP_IN.C FROM EXIM 4.32 SOURCE>
/* These commands need not be synchronized when pipelining */
MAIL_CMD, RCPT_CMD, RSET_CMD,
</START OF SMTP_IN.C FROM EXIM 4.32 SOURCE>
Not that it much matters but SOML FROM and SAML FROM could be added for
full support of the RFC (that is, if exim where to support SOML and SAML
FROM).
<START OF SMTP_IN.C FROM EXIM 4.32 SOURCE>
/* I have been unable to find a statement about the use of pipelining
with AUTH, so to be on the safe side it is here, though I kind of feel
it should be up there with the synchronized commands. */
AUTH_CMD,
</START OF SMTP_IN.C FROM EXIM 4.32 SOURCE>
AUTH_CMD MUST be synchronized, and it is covered in RFC 2920.
Section 3.1 Paragraph 2
<START OF RFC SNIPET>
Additional commands added by other SMTP extensions may only appear as
the last command in a group unless otherwise specified by the
extensions that define the commands.
</START OF RFC SNIPET>
SMTP AUTH is defined as an "SMTP extension" and so would fall under the
above statement and be treated the same as EHLO, DATA, VRFY, etc.
<START OF SMTP_IN.C FROM EXIM 4.32 SOURCE>
/* I'm not sure about these, but I don't think they matter. */
QUIT_CMD, HELP_CMD,
</START OF SMTP_IN.C FROM EXIM 4.32 SOURCE>
QUIT_CMD is specifically defined the same as EHLO, DATA, etc..
Section 3.1 Paragraph 1
<START OF RFC SNIPET>
The EHLO, DATA, VRFY, EXPN, TURN, QUIT, and NOOP commands can only appear
as the last command in a group
</START OF RFC SNIPET>
As to HELP, it is not covered at all, as it is not mentioned as a
command that can be used any just anywhere, and it is a part of the
original 821 definition of commands, so I am with Philip, it does not
matter, but I would move it to the "MAIL_CMD, RCPT_CMD, RSET_CMD,"
group of commads as exim does not really use the HELP command as it is.
<START OF SMTP_IN.C FROM EXIM 4.32 SOURCE>
/* The DATA command is legal only if it follows successful MAIL FROM
and RCPT TO commands. However, if pipelining is advertised, a bad DATA
is
not counted as a protocol error if it follows RCPT (which must have
been
rejected if there are no recipients.) This function is complete when a
valid DATA command is encountered.
Note concerning the code used: RFC 2821 says this:
- If there was no MAIL, or no RCPT, command, or all such commands
were rejected, the server MAY return a "command out of sequence"
(503) or "no valid recipients" (554) reply in response to the
DATA command.
The example in the pipelining RFC 2920 uses 554, but I use 503 here
because it is the same whether pipelining is in use or not. */
case DATA_CMD:
if (!discarded && recipients_count <= 0)
{
if (pipelining_advertised && last_was_rcpt)
smtp_printf("503 valid RCPT command must precede DATA\r\n");
else
done = synprot_error(L_smtp_protocol_error, 503, NULL,
US"valid RCPT command must precede DATA");
break;
}
</START OF SMTP_IN.C FROM EXIM 4.32 SOURCE>
This all looks good to me too, as RFC 2920 only extends the SMTP
protocol, not redefine it, and according the RFC's that 2920 adds to,
they clearly state that some commands SHOULD/MUST be given in a certain
order BEFORE some other commands can be given.
> > things that the server must clearly NOT buffer, and the rules on when
> > to send responses to the client is very clear also.
>
> The fact that the server "doesn't buffer" is neither here nor there.
Yes it does, if the server does not buffer the commands properly, then
the server/client will get "out of sync" and thus cause errors, or
potential errors.
> > Look at Section 3.2:
> [irrelevant, snipped]
> > > Now, this means that it's quite reasonable to suggest pipelining
> > > violations in the situations where no pipelining has been advertised,
> > > but if it has, then it appears that the client can do pretty much
> > > what it wants (within the standard).
> >
> > No it cannot. If the client sends an 'ONLY LAST COMMAND' (EHLO or DATA
> > for example) then it MUST wait for all responses to the command group
> > to be returned, as the client MUST check all of the return codes before
> > sending the next command group.
>
> Quote, please.
Section 3.1 Paragraph 1 and 4
<START OF RFC SNIPET>
P.1
The EHLO, DATA, VRFY, EXPN, TURN,QUIT, and NOOP commands can only appear
as the last command in a group since their success or failure produces a
change of state which the client SMTP must accommodate.
P.4
Client SMTP implementations that employ pipelining MUST check ALL
statuses associated with each command in a group.
</START OF RFC SNIPET>
> > > Other things about this standard appear to be totally braindead,
> > > (VRFY causing a change of state - you what!?) so it's not clear that
> > > a client can't believe that it's allowed to send other things,
> > > because of the unclearness between what goes in one TCP send
> > > operation and the places where it is pausing for a response.
> >
> > There is nothing unclear about it, the client MUST check ALL statuses
> > for each command in the group.
>
> Yes, but it doesn't say that it MUST do that before sending another
> group.
Yes it does.
Section 3.1 Paragraph 1 and 4
<START OF RFC SNIPET>
P.1
The EHLO, DATA, VRFY, EXPN, TURN,QUIT, and NOOP commands can only appear
as the last command in a group since their success or failure produces a
change of state which the client SMTP must accommodate.
P.4
Client SMTP implementations that employ pipelining MUST check ALL
statuses associated with each command in a group.
</START OF RFC SNIPET>
If the client does not wait to process to ALL status codes, then sending
the next batch of commands could cause, may cuse, can cause, will cause
processing errors for either the server and/or the client. Also, take a
look at the examples that are given in RFC 2920, the examples give
proper SMTP transaction use, i.e. the examples follow all of the rules
outlined in RFC's 821/2821.
> > > In particular, if one does several send operations one after the
> > > other, but without waiting for a response (having switched off Nagle,
> > > of course), then it would appear to be conformant to STD60 as
> > > specified,
> >
> > Nope, this is just not true, on or off the client MUST wait for all
> > responses to commands given in a command group before continuing.
>
> Quote, please.
Section 3.1 Paragraph 1 and 4
<START OF RFC SNIPET>
P.1
The EHLO, DATA, VRFY, EXPN, TURN,QUIT, and NOOP commands can only appear
as the last command in a group since their success or failure produces a
change of state which the client SMTP must accommodate.
P.4
Client SMTP implementations that employ pipelining MUST check ALL
statuses associated with each command in a group.
</START OF RFC SNIPET>
> > > but would cause a synchronization error in exim.
> >
> > As it should.
>
> Your opinion,
No, not my opinion, just an understanding of not only how this RFC works,
but also an understanding of how the supporting refernces and network
protocols work.
> not what is said in the RFC. (It so happens that it is my
> opinion too, but I'm curious about this RFC)
Then re-read RFC 2920 AND the references specified there in:
RFC 821
RFC 1123
RFC 1854
RFC 1869
RFC 2197
> > Let us finish looking at Section 3.1...
> > <START OF RFC SNIPET>
>
> [blocking causes deadlock if you don't respect the windowsize]
>
> > </START OF RFC SNIPET>
> > I believe that the 'Failure to perform this check can lead to deadlock
> > conditions' answers your question, if it does not wait, then the client
> > would not be conformant to the standard, thus, causing a
> > 'synchronization error'.
>
> Rubbish.
Not rubbish, it MUST respect the windowsize and it MUST respect the
proper order of commands and the timeing in when to send them, if not,
as I said above:
If the client does not wait to process to ALL status codes, then sending
the next batch of commands could cause, may cuse, can cause, will cause
processing errors for either the server and/or the client.
One such example is a 'deadlock' situation.
> > > So, is it a bug for exim to say "synchronization error" in those
> > > other situations?
> >
> > I would say no, even if what exactly a 'group' is can be fuzzy, when a
> > group starts, and when a group ends is not.
>
> But it's fuzzy because it talks about it in terms of TCP send operations,
> not as pausing to wait for responses.
TCP IS A STATEFULL PROTOCOL! A part of being a STATFULL protocl means
that you wait for responses and acknowledgements to what was sent and
this can apply at more than one level of the application layer. What
you keep wanting to assert would be 'more true' if the SMTP transaction
was done over UDP, and all the client did was send data and forget about
any confirmation of the data being recieved.
RFC 2920 is just an extension to RFC's 821/2821, as such this still
applies to any command given by the client, with the exception that one
or more commands may be given in proper sequence at one time before the
client waits for the necessary return codes rather than one at a time.
RFC 2821, Section 2.1 Paragraph 7
<START OF RFC SNIPET>
The server responds to each command with a reply; replies may
indicate that the command was accepted, that additional commands are
expected, or that a temporary or permanent error condition exists.
Commands specifying the sender or recipients may include server-
permitted SMTP service extension requests as discussed in section
2.2. The dialog is purposely lock-step, one-at-a-time, although this
can be modified by mutually-agreed extension requests such as command
pipelining [13].
</START OF RFC SNIPET>
All that is being modifed by RFC 2920 is again, that one or more
commands may be given in proper sequence at one time before the
client waits for the necessary return codes rather than one at a time.
> > MUST END A GROUP:
> > EHLO
> > DATA
> > VRFY
> > EXPN
> > TURN
> > QUIT
> > NOOP
> > any command added by other SMTP extensions, i.e. AUTH amongst others
>
> You will notice VRFY and EXPN in there... What change of state of the
> server do you think these produce?
SMTP pipelining requires a client that has much more understanding than
a non-pipelining client. It does not matter what 'change of state of the
server' happens at those points, as it is a matter for the client to
handle, not the server.
Section 3.2 Paragraph 2
<START OF RFC SNIPET>
The overriding intent of these server requirements is to make it as
easy as possible for servers to conform to these pipelining
extensions.
</START OF RFC SNIPET>
> > With that definition of when a group ends (therefore anything else
> > starts a group), it is quite clear that the client MUST wait for ALL
> > responses to ALL commands before continuing with the next group.
>
> *blink*
>
> Because elephants are grey, the ocean must be blue.
>
> Non sequitur.
>
> Please feel free to quote the relevant parts of 2920 to back up your
> assertions where I've put "Quote, please." Because those are *exactly*
> the bits it doesn't make clear.
And just to finish up lets look at the language that the RFC uses in the
command use examples.
Section 4
<START OF RFC SNIPET>
Consider the following SMTP dialogue that does not use pipelining:
..
The client waits for a server response a total of 9 times in this
simple example. But if pipelining is employed the following dialogue
is possible:
..
The total number of turnarounds has been reduced from 9 to 4.
The next example illustrates one possible form of behavior when
pipelining is used and all recipients are rejected:
..
The client SMTP waits for the server 4 times here as well. If the
server SMTP does not check for at least one valid recipient prior to
accepting the DATA command, the following dialogue would result:
</START OF RFC SNIPET>
If you actually look at the examples, they specifically show that the
client waits before continueing with the SMTP transaction.
How is this not clear that the client has periods in time that it MUST
wait for the responses from the server?
--
--EAL--
