On Mon, Apr 26, 2004 at 01:25:24AM -0500, Edgar Lovecraft wrote:
> Matthew Byng-Maddick wrote:
> > On Thu, Apr 22, 2004 at 10:54:12AM -0500, Edgar Lovecraft wrote:
> > > Matthew Byng-Maddick wrote:
> > > > While looking at PIPELINING support within SAUCE last night, the
> > > > following issues were highlit by Ian Jackson. The RFC2920 (STD 60)
> > > > spec for SMTP command pipelining includes no precise definition of a
> > > > "command group",
> > > How so? the RFC does not define 'what exactly is in a group', it does
> > > however define quite clearly what commands ARE ONLY last command, no
> > > matter what is in the group, these commands ARE ALWAYS defined as last.
> > Imagine that I send (in one TCP command) all the commands in a group. It
> > doesn't *actually* say I need to wait for an answer.
> Section 3.1 Paragraph 1 and 4
> <START OF RFC SNIPET>
> P.1
> The EHLO, DATA, VRFY, EXPN, TURN,QUIT, and NOOP commands can only appear
> as the last command in a group since their success or failure produces a
> change of state which the client SMTP must accommodate.
>
> P.4
> Client SMTP implementations that employ pipelining MUST check ALL
> statuses associated with each command in a group.
> </START OF RFC SNIPET>
> You must wait. Also, if you are suggesting that your 'group' also
> contains the actual DATA of the message then your client is broken.
You really haven't read or understood what I'm saying, have you? As such,
most of the rest of your attempted justification for what you believe is
basically useless.
What you have quoted (and snippet is spelled with two 'p's), does not say
"You must wait". Anywhere. The fact that the "[c]lient SMTP MUST check ALL
statuses" is in order to make sure that you know what statuses relate to
which command. What that second paragraph explicitly *doesn't* say is:
| Client SMTP implementations that employ pipelining MUST check ALL
| statuses associated with each command in a group before sending the
| next group.
> Section 3.1 Paragraph 4
> <START OF RFC SNIPET>
> If the DATA command was properly rejected the client SMTP can just
> issue RSET, but if the DATA command was accepted the client SMTP
> should send a single dot.
> </START OF RFC SNIPET>
> If your client does not wait to check all status codes as it MUST do,
> then how does your client know to send a single dot or not?
That is not what it MUST do, as I've just pointed out above. The omission
of the last four words from the paragraph you quoted means that you cannot
assert what you have just asserted. Also note that the single dot is a
SHOULD not a MUST.
> > CAN ONLY is not part of RFC2119, and the word MUST does not appear in
> > that text. It seems pretty clear until you actually look at the language,
> > which doesn't prohibit the scenario I've outlined above. The RFC is
> > confused, and full of layer violations (expecting the Layer 4 stuff to
> > know about what's going on at Layer 3).
> Yes it does.
> Section 3.1 Paragraph 1
> <START OF RFC SNIPET>
> The EHLO, DATA, VRFY, EXPN, TURN, QUIT, and NOOP commands can only
> ^^^^^^^^
> appear as the last command in a group since their success or failure
> </START OF RFC SNIPET>
> Unless I do not know my English, "can only" means that those commands
> are ONLY allowed to be used as a LAST command in a command group. As I
We're not talking about English, we're talking about a specification
document, and a standard, with conventions on the choice of words, provided
by RFC2119, in order to ensure complete unambiguity. As written, this
document is completely ambiguous. This is my complaint with it.
[irrelevant tat that I am fully aware of, and have made clear that I'm
aware of many times already]
> case, when a client ends a command group with any command it MUST check
> ALL statuses associated with each command in that group.
Yes, but this does *NOT* force it to wait to send the next group.
> > Yes, but at no point does it explicitly say that a command group
> > terminator MUST wait for a response. Please point out the part where it
> > says this. It only says that it must be the end of a TCP send operation.
[the same quoted text of the RFC above]
> If your client does not wait, then how can it check 'ALL statuses' as it
> MUST do?
Errr? By reading the status codes, as you'd expect.
> <INSERT RESPONSE FROM TONY>
> > It's implicit in the state-transition business, e.g. the way the client
> > has to handle DATA, and the mention that NOOP is a synchronization point.
> > Yes it could be clearer.
> </INSERT RESPONSE FROM TONY FINCH>
> I would only agree that it could be clearer if I did not have the
> understanding that RFC compliance is more than just understanding any
> ONE RFC. It all seems very clear to me.
I'm looking at it in reference to several others, and it seems very *unclear*
about the whole concept of waiting before sending the next group. In
particular, it doesn't help that the DATA state transition is mentioned
at the same time as a VRFY or EXPN "state transition". The fact that these
basically don't exist seems to cast doubt on the entire meaning of the group
to me.
> > > > and in particular, although it's implied that the "last command
> > > > of a command group" is one that should wait for replies, the server:
> > > Not just implied, the client MUST wait for all replies.
> > Quote, please.
[ Edgar quoted the same text that does *NOT* say what he think it says,
again ]
> > > > | (9) MUST NOT flush or otherwise lose the contents of the TCP input
> > > > | buffer under any circumstances whatsoever.
> > > That is to ensure that the server has seen the entire 'command group'
> > > sent by the client, it has nothing to with the client not waiting for
> > > ALL responses to the 'command group' which the client MUST do. There
> > > are also
> > Quote, please.
[ and again. I love the way this goes on... ]
> > It may be "to ensure...." however, if you think through what is
> > happening, because there's no restriction on the command group actually
> > having to wait for a reply (despite your assertions of a MUST, it doesn't
> > exist), it also means that you can't legally (as a server) throw away
> > stuff that exim currently considers to be a pipelining violation.
> Please explain what things exim currently throws away and considers to
> be pipelining violations that you do not agree with.
554 Synchronization error, connection closing. (note that this will NOT
provide status codes for any following commands...)
[irrelevant quoted source and assertion]
> > > things that the server must clearly NOT buffer, and the rules on when
> > > to send responses to the client is very clear also.
> > The fact that the server "doesn't buffer" is neither here nor there.
> Yes it does, if the server does not buffer the commands properly, then
> the server/client will get "out of sync" and thus cause errors, or
> potential errors.
No, you'll get a deadlock.
> > > No it cannot. If the client sends an 'ONLY LAST COMMAND' (EHLO or DATA
> > > for example) then it MUST wait for all responses to the command group
> > > to be returned, as the client MUST check all of the return codes before
> > > sending the next command group.
> > Quote, please.
[once again, Edgar quotes the _SAME_ thing that still doesn't say what he
thinks it says]
> > > > Other things about this standard appear to be totally braindead,
> > > > (VRFY causing a change of state - you what!?) so it's not clear that
> > > > a client can't believe that it's allowed to send other things,
> > > > because of the unclearness between what goes in one TCP send
> > > > operation and the places where it is pausing for a response.
> > > There is nothing unclear about it, the client MUST check ALL statuses
> > > for each command in the group.
> > Yes, but it doesn't say that it MUST do that before sending another
> > group.
> Yes it does.
[RFC2920:]
> Client SMTP implementations that employ pipelining MUST check ALL
> statuses associated with each command in a group.
Where does it say that!?
You've now quoted this same thing 5 times in this message, and it still
doesn't say explicitly what you're claiming it says. This is a standards
document, not some wishy-washy, I-feel-like-interpreting-it-in-this-way
document. The language really should be totally unambiguous and all cases
should be covered.
> If the client does not wait to process to ALL status codes, then sending
> the next batch of commands could cause, may cuse, can cause, will cause
> processing errors for either the server and/or the client. Also, take a
> look at the examples that are given in RFC 2920, the examples give
> proper SMTP transaction use, i.e. the examples follow all of the rules
> outlined in RFC's 821/2821.
These examples are just that, examples. They do not say what should happen
at the edge case that I've brought up. Moreover, the quoted text that you've
presented me with 5 times does not say what you claim it says, even if that
is the *spirit* of it (which, as I said, I happen to think that it is).
However, standards documents should have spirit and letter being equal, and
this does not seem to be true with RFC2920, and this thread has arisen from
the fact that it seems that it would be perfectly legal (in the letter) to
interpret it in a different way, that command groups are not the elements
that should now be in lock-step in the protocol (which is, ultimately, the
effect of what we're arguing). Your incessant requoting of the same text
doesn't help confirm that that's what it means, and nowhere in the document
(unfortunately) does it actually say anything that fully backs up your
position.
> > > > In particular, if one does several send operations one after the
> > > > other, but without waiting for a response (having switched off Nagle,
> > > > of course), then it would appear to be conformant to STD60 as
> > > > specified,
> > > Nope, this is just not true, on or off the client MUST wait for all
> > > responses to commands given in a command group before continuing.
> > Quote, please.
[Edgar excels himself in his use of copy and paste]
Once again, you've quoted the same text that does not say what you claim it
does. It says the client must check all statuses, it does not say that it
must do this before doing anything else. There is a huge difference in these
two things.
> > > > but would cause a synchronization error in exim.
> > > As it should.
> > Your opinion,
> No, not my opinion, just an understanding of not only how this RFC works,
> but also an understanding of how the supporting refernces and network
> protocols work.
I too, believe that I have an understanding of this, and while my initial
reading was the same as yours, I've realised that actually it doesn't say
what I thought it said when I first read it.
> > not what is said in the RFC. (It so happens that it is my
> > opinion too, but I'm curious about this RFC)
> Then re-read RFC 2920 AND the references specified there in:
> RFC 821
> RFC 1123
> RFC 1854
> RFC 1869
> RFC 2197
Depressingly, 2119 is not listed as a reference, or maybe we'd have some
properly unambiguous language.
1854 and 2197 are just earlier versions of 2920, although it references
them, you cannot use them for argument of what 2920 means, as it is 2920
that is STD60, not either of the above, it should, really, be a document
which draws on its references. 821 and 1123 provide context for the SMTP,
and 1869 is part of STD10 which defines the SMTP service extension
additions to SMTP.
> > > Let us finish looking at Section 3.1...
> > > <START OF RFC SNIPET>
> > [blocking causes deadlock if you don't respect the windowsize]
> > > </START OF RFC SNIPET>
> > > I believe that the 'Failure to perform this check can lead to deadlock
> > > conditions' answers your question, if it does not wait, then the client
> > > would not be conformant to the standard, thus, causing a
> > > 'synchronization error'.
> > Rubbish.
> Not rubbish, it MUST respect the windowsize and it MUST respect the
> proper order of commands and the timeing in when to send them, if not,
> as I said above:
You missed the whole point of that paragraph. It is perfectly possible to
implement a client and server which don't block, and which therefore don't
need to care about the underlying window size.
> If the client does not wait to process to ALL status codes, then sending
> the next batch of commands could cause, may cuse, can cause, will cause
> processing errors for either the server and/or the client.
> One such example is a 'deadlock' situation.
You could only get a deadlock ON THE CLIENT if you are doing blocking reads
or writes and trying to send too much data. The deadlock should not occur
on the server end if it is implementing 2920 to spec.
Therefore, you can simply get round this, and still not have to wait "to
process ALL status codes" which is a quote, out of context as it explicitly
DOES NOT say "wait", by using non-blocking I/O. Which is exactly what that
bit of 2920 says.
> > > > So, is it a bug for exim to say "synchronization error" in those
> > > > other situations?
> > > I would say no, even if what exactly a 'group' is can be fuzzy, when a
> > > group starts, and when a group ends is not.
> > But it's fuzzy because it talks about it in terms of TCP send operations,
> > not as pausing to wait for responses.
> TCP IS A STATEFULL PROTOCOL! A part of being a STATFULL protocl means
> that you wait for responses and acknowledgements to what was sent and
> this can apply at more than one level of the application layer. What
Errrr.... You what!?
TCP is stateful, (one 'l', please), which is all about the stream not
coming in out of order, and knowing what the state of the connection is.
This has nothing to do with what data you send across the stream. It doesn't
have anything to do with how you implement your application layer at all.
(think, if you will about half-closes).
If what you're trying to say above were actually true, then the entirety of
RFC2920 wouldn't be possible.
> you keep wanting to assert would be 'more true' if the SMTP transaction
> was done over UDP, and all the client did was send data and forget about
> any confirmation of the data being recieved.
No, the confirmation of the data being received ('i' before 'e' except after
'c', if the sound is 'ee') is in order to make sure that the stream comes in
reliably, the application gets no explicit confirmation that that data has
been received. TCP allows one to let the kernel do the work, so the
application can be fire-and-forget.
> RFC 2920 is just an extension to RFC's 821/2821, as such this still
> applies to any command given by the client, with the exception that one
> or more commands may be given in proper sequence at one time before the
> client waits for the necessary return codes rather than one at a time.
Yes, I agree with this. However, it doesn't disagree with anything that
I've said, or anything that you've said.
[irrelevant rfc2821 quote, snipped]
> All that is being modifed by RFC 2920 is again, that one or more
> commands may be given in proper sequence at one time before the
> client waits for the necessary return codes rather than one at a time.
Yes. Again, this doesn't disagree with anything that I've said.
> > You will notice VRFY and EXPN in there... What change of state of the
> > server do you think these produce?
> SMTP pipelining requires a client that has much more understanding than
> a non-pipelining client. It does not matter what 'change of state of the
> server' happens at those points, as it is a matter for the client to
> handle, not the server.
This is true, but not an answer to what I said.
[irrelevant RFC snippet snipped]
> > > With that definition of when a group ends (therefore anything else
> > > starts a group), it is quite clear that the client MUST wait for ALL
> > > responses to ALL commands before continuing with the next group.
> > *blink*
> > Because elephants are grey, the ocean must be blue.
> > Non sequitur.
> > Please feel free to quote the relevant parts of 2920 to back up your
> > assertions where I've put "Quote, please." Because those are *exactly*
> > the bits it doesn't make clear.
> And just to finish up lets look at the language that the RFC uses in the
> command use examples.
[example text snipped]
Examples are not definitions.
> If you actually look at the examples, they specifically show that the
> client waits before continueing with the SMTP transaction.
The fact that the client waits in those examples does not dispute my
reading of the language in the RFC.
consider the following sentences, which do not appear in the RFC, but
would allow both those examples, and what I've already said.
"The client MAY wait for the server to send it all status codes before
sending a new command group."
or even:
"The client SHOULD wait for the server to...."
> How is this not clear that the client has periods in time that it MUST
> wait for the responses from the server?
Where in 2920 does it say "MUST wait"?
It doesn't say that. It defines a command group as being something that you
send in one TCP packet. It says the server must never throw away input
buffer, and it does NOT stop you having more than one command group in flight
at any one time. This is not forcing you to wait for responses before carrying
on, though, as Tony has rightly pointed out, you pretty much have to for DATA.
MBM
--
Matthew Byng-Maddick <mbm@???> http://colondot.net/
(Please use this address to reply)
