Phil Pennock wrote:
> On 2009-05-07 at 03:43 +0800, W B Hacker wrote:
>> - IF I were to receive incoming purporting to be from, using your
>> example, globnix.com AND the rDNS passed muster, I'd presume you
>> intended to send, and - while making other tests - not have a care as to
>> the presence, let alone 'nuances' of an spf record.
>>
>> Simply put, it tells me nothing any more useful than what is already in
>> front of me.
>
> No. If the rDNS is for IP address space allocated by AfriNIC to a
> company in Nigeria, then you have no reason to suspect that the mail is
> potentially dodgy other than by content examination. The public
> assertion that "this domain does not send mail" provides you with a data
> point.
>
> Whether or not you use that data point is up to you.
Of course. But the likelihood that the very sort of arrival you mention
would ever be in the class of folks 'polite and professional' enough to
tell us - by spf or otherwise - that they 'never sent' mail would be
vanishingly small.
By definition, the arrivals 'of concern' are not from 'the good guys'.
Having an 'extra' confirmation tool from those who *are* 'good guys' is
warm and fuzzy - but they were not the problem in the first place.
So I really am back to what was 'already in front of me'.
The extra test process, meanwhile has used at lest a few fractional ergs
of energy and delayed some CPU time-slices that could have gone toward
tests that are more 'profitable' in avoiding the miscreants.
NB: I'm not a 'green' per se, but we are up against the wall on UPS
budget, and trying to do with, for example, VIA C7 what we once did with
Intel multi-core.
>
>> - IF, OTOH, said arrival was a forgery, I'd not need to look at an spf
>> record to determine that, either.
>
> You have a magic gateway which detects forgeries sent from static IP
> hosts with valid rDNS. I am impressed.
>
Not magic - but it does have some other tests so controversial I'll not
go into them on this thread.
>> - A 'legit' message would live or die on the credentials of the sending
>> servrr, (rDNS, FQDN in HELO, correct format and MIME-type usage), AND
>> NOT ClamAV or SA finding malware or unwanted content/attachments.
>>
>> - A forgery would not make it past acl_smtp_connect.
>
> The majority of forgeries, perhaps; but not all forgeries are so
> trivially and obviously wrong.
True - one of the current challenges is Spam/malware crafted by experts
who have studied and applied everything discussed here and on anti-spam
lists. Still not common, but *very* professionally crafted to get under
all manner of radar.
But those folk will not be publishing records to help us avoid them.
Quite the reverse - they will be trying to publish that which increases
our 'trust'.
> I don't care so much about the 90% of
> spam that is trivially rejected by such checks, I do care about the 10%
> of spam which makes it past those and yet still comes in sufficiently
> large volume that filtering it is desirable.
>
> It's all about providing signal. I publish signal, you're free to use
> or ignore that signal.
>
Appreciated.
>> Are you making this sort of callout / DNSDB lookup on all - or even a
>> large percentage of traffic transiting?
>>
>> Surely the percentage of arrivals that might have usable information
>> must be small?
>
> "Callout" is somewhat disingenuous in a mail context, unless you're
> saying that looking up any records in DNS is a callout, given the
> negative connotations to SMTP callouts or other such heavyweight checks.
>
Thinking here in terms of resource load (both ends) and response time
(remote OR local DB or cache).
Ex: I make use of 'remote' dynamic-IP RBL checks. But also build stats,
such that over time, the 'chronic' hits can be moved first to an Exim
lookup, later to a firewall rule. Lowers the load at my end AND the RBL
provider.
Also means those so tagged don't get de-listed here even if de-listed on
sorbs.
Life is like that sometimes...
:-)
> The cost of a DNS lookup which fails is fairly light, except in the case
> of badly broken DNS setups (timeouts on requests for RR types which are
> not configured, SERVFAIL, whatever) and anyone sending mail should
> expect a lookup for an SPF record in any case. If there's an SPF
> record, I've populated my cache for some SPF analysis stats I am
> considering running. If it lets me reject mail, that's even better.
>
> In terms of how much benefit is seen in implementing this: over the past
> month this has been responsible for between 0.1% and 2.8% of my daily
> rejections. This is on a server which just handles mail for myself and
> my wife -- I'm no longer a paid postmaster (and no longer working on
> anything related to email, either).
>
Add a few old friends and fellow-travelers, and much the same here since
I retired.
> Peaks are (file, percentage, SPF rejections, total rejections):
> rejectlog-20090403: 2.842 32 1126
> rejectlog-20090421: 0.800 45 5622
>
> I use clamav but don't currently do content-based rejections, as I get
> few enough spam which make it through the other checks that I can
> discard the remainder fairly safely. I don't have the skill or time to
> set up decent machine-learning content analysis to the point where it
> doesn't upset my paranoid side. By "discard", I mean "save it into a
> spam folder" so that at least I'll have a nice *cough* large corpus by
> the time I next get to spend some serious time on this.
>
ACK. Per-user 'Suspect' folders (multi-level if asked for) are
auto-created by Exim in the in IMAP mailstore structure, so even those
who delete the whole folder get a new one at next such 'hit'.
SA is *so* stripped-down here it's coders would scream.
But we ask of it ONLY what has a high-value payback and has not already
been done inside of Exim. SA is seldom called into action, and runs
blitz-quick (for perl) when it is.
> I consider the cost of a DNS lookup to be less than the cost of
> verifying a DomainKeys/DKIM signature and really to be small enough
> that, at my current scale, it's worthwhile.
Agree that. In 'early days' we actually *added* spamint points on header
'claim' of DK / DKIM or hashcash. Now we ignore all.
We still do a 'hard fail' on any that have the 'licensed to spam'
credentials.
> Of course, DNS caching
> capacity scales differently to CPU scaling, so this would obviously need
> to be evaluated differently at mega-large providers. For most of us, a
> DNS cache or two will have more than adequate resources.
>
ACK.
> Of course, I reject immediately, rather that setting an ACL variable so
> I can deny at the end of the ACL but have more complete stats by knowing
> how many different checks would have matched, so some of these would
> have been rejected by other methods too. Reworking this to provide me
> with better stats is deferred until the next occasion that I have
> sufficient free time.
>
> -Phil
>
..still working on it now. Perhaps 80% of my complex acl's can go away
now that I no longer have clients who 'must never lose' a potential
client inquiry.
What we are coming to for a 'friends and family' environment essentially
turns the traditional 'be generous in what you accept' process upside-down:
"If in doubt - reject, and right up-front."
Somebody's, sister, old mate, grandchild, golf club, or alumni
association will mention the problem to them by phone, fax, or Gmail,
and we'll add a VIP or Whitelist entry 'at once'..
...then also advise them what (usually minor) change can be onpassed to
their correspondent's provider that could improve their probability of
getting through - to others as well as ourselves.
As more mailadmins globally are getting stricter, most now go off and
confirm that advice, as they will have had other rejections - then,
thankfully, more often apply it than they might once have done.
Result: Exception list 'hits' drop, and it can be pruned - a mere 19
entries at the moment.
So the good news is that 'the good guys' *are* improving their environment.
The bad news is that such exploit attempts as once moved primarily over
smtp have now moved to poisoned web pages... a whole 'nuther battlefront...
:-(
Bill
