On Fri, 24 Sep 2004, Tony Finch wrote:
> On Thu, 23 Sep 2004, Mark Moseley wrote:
> >
> > Ironically, my best numbers are from HELO checks
>
> Our best rejecter is the RBL+ (35% of rejections) followed by sender
> callback verification (25%) and the SBL-XBL (15%).
Of course, the problem with reporting these kinds of statistics are
that it depends strongly on what order the tests are carried out.
If you're awarding spam-rating points rather than outright rejection,
then sure, you can later evaluate which criteria were or were not
triggered. But for tests which lead to an outright rejection, one
doesn't really know whether a later test would - if given the chance -
have triggered or not.
We've moved, over time, towards a sequencing of steps which, roughly
speaking, optimise these principles:
1. Steps which can be carried out cheaply (that means basically tests
which use local blacklists; or which rely on lookups that we're going
to perform anyway, such as identd checks or reverse DNS on the calling
IP) are done first. Relatively cheap external lookups such as DNSrbls
come next. More-expensive lookups such as selective callouts of the
envelope-sender address come late.
2. As a secondary principle, if we can avoid exposing too much useful
information then we'll sequence the testa accordingly, e.g we'd rather
tell the caller that their calling IP is blacklisted than to tell them
that the RCPT TO address does not exist (to reduce the prospects for
dictionary attacks and list-washing).
Of course, they have to get past all of that before they're considered
for antivirus tests and, finally, spam-rating.
But, as I say, the ordering of the tests does skew the statistics
rather strongly. Many of the mails which we reject, if they hadn't
been rejected on one of the earlier criteria, would still have been
rejected later; but we don't actually know that just by looking at the
server logs.
Coming back to the topic: here's an example of the kind of thing that
makes me wild. There's an ongoing stream of bounces in Cyrillic,
coming mostly from Russian servers, for abusive mails which faked one
of our addresses as sender. Look at this specimen, from one of the
bounce reports:
We are sorry to notify you that address xpoff@??? does not exis=
t.
Please, correct the address and resend.
And then, in the quoted headers, we see:
Received: from [62.108.162.134] ([62.108.162.134]:32006 "HELO 62.108.16=
2.134"
smtp-auth: <none> TLS-CIPHER: <none> TLS-PEER-CN1: <none>)
by mail.yandex.ru with SMTP id S709319AbUIXJNN (ORCPT
<rfc822;xpoff@???>); Fri, 24 Sep 2004 13:13:13 +0400
Received-SPF: none (mx9.yandex.ru: 62.108.162.134 is neither permitted =
nor denied by domain of astro.gla.ac.uk) client-ip=3D62.108.162.134; en=
velope-from=3D*snipped*@???; helo=3D62.108.162.134;
<rant>
So these idio^Wmisguided souls have somehow got the expertise to go
decoding SPF records, but hadn't got the sense to reject mail for a
non-existent address of theirs at SMTP time, nor apparently the sense
to recognise that they'd have been better-advised not to accept mail
from 62.108.162.134 in the first place - not only because it failed to
present a sensible HELO, and doesn't have reverse DNS, but also
because it's well-listed in a range of DNSbls.
These collateral/backscatter/ricochet (call them what you will)
bounces are a festering nuisance, and almost impossible to block
reliably (particularly as we have to handle bona fide Cyrillic mail
for some of our users). I collected a few dozen of them to see if I
could pick out some common factors to help in rejecting them, but "no
such luck". They came from lots of different - usually .ru - mail
servers, with lots of different patterns of error report and presence
or lack of quoted headers, and presence or lack of quoted body
material from the original spams (which themselves seem to be quite
variable).
</rant>
Quite a proportion of the bounces came from "CommuniGate Pro", but by
no means all of them; and it would seem to be overkill to reject all
bounces merely on the grounds that they were coming from "CommuniGate
Pro".
sigh.
