On Fri, Jul 23, 2004 at 11:09:55AM +0100, Philip Hazel wrote:
> How can Exim tell when it is receiving a "submission"?
If the message has no pre-existing Received: header, then it must be a
submission, musn't it? This is not all-inclusive, I think, but it's a
start...
> 1. For -t it is obvious that it is a submission.
>
> 2. Without -t, it is not (I contend). Consider a message that came from
> outside, was piped to Majordomo or MailMan, and is then being
> re-injected for final delivery. This isn't a "submission", at least
> not from the local host.
Your contention is a fantasy. Though it is unfortunate, many MUAs
(most notably PC-oriented ones) inject their messages into the mail
system via other means, such as connecting directly via SMTP. You
mentioned this yourself, following this last statement.
> > For example, the MTA can re-write headers to ensure that return mail
> > will be deliverable.
>
> It can, but SHOULD or MUST it? RFC 821 says:
My point wasn't whether it should or must, only that other reasons to
do so exist and are allowed by the RFCs, as you yourself point out.
In which case, what's wrong with adding one more reason?
> > However, some message systems may use information from the contents
> > to create the envelope. It is intended that this standard
> > facilitate the acquisition of such information by programs.
> >
> > Here, I think it is clear that the term "message system" refers, at
> > least in part, to the MTA. It also states that the intent of the RFC
> > is to make it easy for the "message system", again the MTA, to create
> > an envelope. This is the purpose of the -t option which you have
> > stated that you don't like.
>
> With respect, understanding "message system" to mean "MTA" (even in
> part) is just one interpretation.
Maybe, but as used here I think it's the only possible interpretation
that makes sense. To suggest that it could only refer to the MUA
seems absurd. The MUA need not aquire the envelope information from
the headers of the message; it knows full well who the intended
recipients are, as specified by the user. In fact I can't think of a
single MUA I've ever seen which doesn't explicitly ask the user for
the recipients seperately from composing the message.
And as a system administrator with many years experience managing mail
systems, the only use of the term "mail system" I have ever seen is
one which encompases, and even primarily refers to, the MTA. Your
statement may be tecnically true; other interpretations exist. But as
a matter of practicality, they are nonsensical.
> > I will include the idea that it must ensure that at most only the
> > recipient named in the Bcc line receives a copy of the messge with a
> > Bcc line mentioning his address (and no others).
>
> Not correct. The Bcc line is permitted to contain a list of ALL the Bcc
> recipients:
>
> "but the recipients on the "Bcc:" line get a separate copy of the
> message containing a "Bcc:" line. (When there are multiple recipient
> addresses in the "Bcc:" field, some implementations actually send a
> separate copy of the message to each recipient with a "Bcc:"
> containing only the address of that particular recipient.)"
Yes, it is. So what? That doesn't preclude the idea that the MTA or
some other agent can do the right thing and weed out recipients who
don't need that information.
You seem to be missing my point, so I'll state it as clearly as I know
how:
Authors of software which is intended to be used by people other
than themselves must sometimes assume responsibilities above and
beyond what are specified by RFCs and/or the law. These
responsibilities come from common sense, and an assumed
responsibility to prevent users from receiving ill as a result of
using their software as intended.
Section 5 of RFC outlines security concerns relating to the various
methods of handling the Bcc line. Three methods are permitted, which
is unfortunate. [Actually it's four; there are two variations of the
second method.] Based on the legitimate security concerns outlined by
the RFC, only one makes sense, which is the behavior I have advocated
here. Failure to address those concerns, while arguably allowable by
the RFCs, may result in potentially serious damage to users of the
software, who have a reasonable expectation that Bcc'd recipients of
their messages will not be revealed to non-Bcc'd users (or, I contend,
anyone else).
I'm saying that common sense and decency mandate that the MTA should
enforce this. Why? Because:
1. it is known that some mailers expect (something like) this behavior.
2. it is known that many MTAs already do (something like) this, so
that expectation is a reasonable one.
3. The MTA is in a position to detect the situation where recipients
who have not been Bcc'd will see the Bcc line.
Since it can do so, it should, whether or not there exists an RFC
which says it should. It is not an obligation dictated by standards,
but by common sense and decency.
> > However, RFC 2822 DOES NOT ALLOW for the recipients named in the To:
> > and the Cc: lines to receive copies of the Bcc: lines.
>
> Sure. But it does not make it clear whose job it is to enforce this.
> That's due to the obscurity of the RFC. I believe it should be enforced
> as early as possible, by the MUA, because of the difficulty an MTA has
> in deciding when it is handling a "submission". For "foreign" messages
> that originate elsewhere an MTA has no right to make this kind of
> change.
Sure, but see above. I don't see why that matters. Ideally, all
parts of the mail system, from the MUA to the MDA should enforce this,
just in case someone else along the chain got it wrong.
> > The fact is, the vast majority of other MTAs DO strip the Bcc lines,
>
> Under what circumstances? Always?
As nearly as I can tell, yes. At least by default. I did some tests
with sendmail today where I generated messages from a) my mailer, b)
from the sendmail command line, and c) by telnetting directly to a
sendmail server and typing in SMTP commands and messages manually. In
all cases, it stripped the Bcc header from the message. I even
included a received: header to attempt to see if this would make a
difference... It didn't.
There's also this in the sendmail manual for 8.13:
6.3.1. Built-in Header Semantics
[SNIP]
H_STRIPVAL Strip the value from the header (for
Bcc:).
[snip]
Let's look at a sample HdrInfo specification:
struct hdrinfo HdrInfo[] =
{
/* originator fields, most to least significant
* */
"resent-sender", H_FROM,
"resent-from", H_FROM,
"sender", H_FROM,
"from", H_FROM,
"full-name", H_ACHECK,
"errors-to", H_FROM|H_ERRORSTO,
/* destination fields */
"to", H_RCPT,
"resent-to", H_RCPT,
"cc", H_RCPT,
"bcc", H_RCPT|H_STRIPVAL,
Though, in the actual code in sendmail/conf.c, this H_STRIPVAL seems
to have been replaced by H_BCC. I can only assume that it does the
same thing, as it is not actually mentioned in the manual, and I
haven't read through all the code (and have no intention of doing so).
I'm satisfied that what's in the manual seems to reflect what actually
happens in empyrical testing.
> > The point is, Exim's behavior is a departure from what the rest of the
> > world does and expects,
>
> If that is true, I am prepared to consider changes, BUT: (a) I need to
> be convinced that it is true and (b) there needs to be a PRECISE
> definition of the circumstances under which this action is to be taken.
> I do not consider "always remove any Bcc lines from any message" to be
> good enough.
Well, that seems to be precisely what sendmail does. So you'll have
to figure out for yourself what you consider "good enough."
> > Exim is only deficient in handling #2. The RFC states that this line
> > may exist in the message, and it also states that it is not to be
> > passed on to recipients other than those named in the Bcc line.
>
> But your MUA has just passed it on to those recipients!
With the assumption that the MTA will handle it. As you said
yourself, the RFC doesn't make it clear whose responsibility it is to
address this problem... So it isn't unreasonable for the mailer to
expect that, especially since at least Sendmail is known for certain
to do so, and I'm pretty certain that others (such as Postfix and
qmail) do something similar.
> > Logically, if it a) exists in the message, as pased on by the MUA, and
> > b) must not be passed on to recipients which are unnamed in the Bcc
> > line, then the only place where it is possible for this situation to
> > be remedied is within the scope of the processing done by the MTA.
>
> ... as long as the MTA knows it is receiving a message from a "local"
> MUA in "submission" mode.
I don't agree, but I guess it's sufficient. But I think your
definition of "submission mode" is inadequate. I think you're correct
in assessing that it is difficult to determine, and so the MTA should
ALWAYS behave this way, IF it detects that not doing so will result in
non-Bcc'd recipients seeing the Bcc line. I don't think we should
write software to minimally meet responsibilities stated in standards
documents... I think we should do our best for the user.
> > The best case scenario is for the MTA to ensure that seperate copies
> > of the message are delivered to each of the recipients named in the
> > Bcc line, with a modified Bcc line mentioning only that individual
> > recipient.
>
> That is only one choice. It is also permissible to name all Bcc
> recipients. The MTA does not know which is desired.
It is permissible by the RFC, yes; but let's be realistic. Who uses
Bcc and wants all the recipients to see the Bcc line? No one. That
defeats the purpose of having it in the first place. Otherwise just
use the Cc line.
The RFC doesn't go far enough. It outlines all the security issues
very clearly, but fails to come to the correct conclusion: A user who
has been Bcc'd should see that THEY have been Bcc'd, but no one else
should. That is, each Bcc'd recipient should receive a copy of the
message, with a Bcc line that includes only their address on the Bcc
line. This is the only behavior that makes sense. And from a
practical standpoint, I think the MTA should enforce this.
> And what if the Bcc recipient turns out to be an alias of a mailing
> list? (For example).
So what if it does? The MTA will deliver it to the mailing list host,
and the MDA will deliver it to the subscribers of the mailing list.
The message will contain a Bcc line showing the mailing list address,
and all the recipients will receive a copy that shows it. What's the
problem here? The Bcc line still shows the desired information, which
is how the user came to receive a copy of the e-mail. This is one of
the concerns raised by Sec. 5 of RFC 2822. Presumably, the user knows
what mailing lists they've subscribed to... ;-)
> And worse, if the mailing list contains one of the
> non-Bcc recipients? (OK, perhaps I'm getting carried away here.)
Yes, I think so... ;-)
> > Ok, what you're suggesting here is that the MUA generate individual
> > copies of the message for each Bcc recipient.
>
> Or two copies, one for each set of recipients, if you want all the Bcc
> recipients to be listed to all of them. Or only one copy if you don't
> want any Bcc lines in the message at all. Your choice. The RFC allows
> all 3 formats. Only you know which you want - the MTA doesn't.
But we've already established that the RFC sucks. I would hope that
you have enough pride in your work to make a better MTA than merely
what is required by a poorly written RFC...
> > This certainly is possible, but is not practical. The MTA already
> > does this as a normal part of processing outgoing mail.
>
> Not unless it's qmail. If all recipients are routed to the same remote
> MTA, Exim will send (by default) ONE copy, as suggested by RFC 2821.
Yes, but when there are many recipients, the MTA must generate
multiple messages. This was my point. It already can and does send
multiple copies when it needs to do so. Most of the code already
exists. So why not add this functionality in the same place? Rather
than duplicating all that work in all the MUAs which exist, why not do
it in the MTAs, which already need to do some similar processing
anyway? From a practical standpoint, the MTA is the place to do this.
> > If this argument doesn't convince you, then I expect nothing will, so
> > I'll sign of here.
>
> I'm not convinced. However, I have a record of being persuaded on other
> issues if enough people have a consensus. I suggest:
Well, I'm done here... I don't use Exim, and I don't expect to any
time soon, so I've spent enough time on this here. But I do think
that RFC 2822 doesn't go far enough to protect Bcc recipients and
their sender, and I will probably post on this on Bugtraq in the very
near future. Exim is the only MTA which I have been able to
specifically identify as not doing anything with Bcc lines, which is
why I have spent time on it here. I will post more general comments
on Bugtraq. But thanks for your time discussing it with me, it helped
me to solidify a lot of thoughts about the issues.
Cheers.
--
Derek D. Martin
http://www.pizzashack.org/
GPG Key ID: 0x81CFE75D
