(CCed to exim-users since this topic is interesting)
Tim Jackson wrote:
>>>So, does this mean that if Exiscan detects broken MIME (and
>>>exiscan_demime_action = pass), it just doesn't unpack it at all? This
>>>is a bit worrying, at least until Tom thinks it's robust enough to
>>>turn exiscan_demime_action = reject on.
>>
>>The -24 is already very robust, I have just fixed one (final ?) bug with
>>single-part b64 messages (which are very uncommon).
> OK. How about false positives though? My concern is obviously that however
> robust Exiscan is, that there are so many mailers out there that some
> "real" ones are bound to be broken and we may have to "accept" just to
> avoid losing mail
> For example, I once came across a PHP script that someone had written that
> purported to send MIME mail. It *seemed* to work, and indeed in every mail
> client I tested, it worked fine, but it was actually pretty badly broken
> and would probably be rejected by Exiscan. Clearly, the fault here is in
> the broken script, but who's to say how many dodgy PHP/Perl/etc. scripts
> there are out there being used unwittingly for genuine purposes...
I think the false positive rate should be reasonably low. I have tested
my demimer with ~3000 messages from my own INBOX. I get a lot of mail
from very different sources, sent with very different mailers. All of
these mails non-spam, non-virus legit messages, lots of them with
attachments. Out of this testset, only 4 messages were rejected as
broken, and they all came from a sloppily programmed Web-Mail gateway,
and all of them had broken quoted-printable encodings. I have relaxed
the quoted-printable check to be more tolerant now, and this brings my
own false-positive rate with NON-virus and NON-spam messages to zero.
> Interestingly, I couldn't see an *attempt* to scan in the clamd logs, (but
> I could be missing something) although it wouldn't have succeeded since
> clamd doesn't appear to support MIME.
I have never used clamd, so I can't say anything on that topic.
Kavdaemon and Sophos have their own MIME engines, and both of them are
less than perfect, too.
I have a test-set here that includes a lot of exploits that malware can
use to bypass mail scanning engines. It is amazing what Outlook and
Oulook Express will do to unpack a broken MIME container.
Zero bytes in the encoded data ? Nevermind, we skip them.
Empty lines in base64 data ? Simply ignored !
Overlog proposed filenames ? Just truncate at ~700 bytes.
Illegal quoted-printable encoding ? Just use a literal '=' !
These are just some examples. The one with the long filename has already
been successfully exploited. The problem is the closed-sourceness of
MIMEOLE, the Microsoft MIME implementation. Not all of the possible
exploits are known, and even working around the known ones is a headache.
My approach is to be as RFC-compliant as possible, while minimizing the
collateral damage. This is the only thing that will keep you safe from
Microsoft MIMEOLE.
> This still leaves the problem that if you're using a non-MIME capable
> scanner (e.g. clamd), unless you turn on strict MIME checking, you're
> going to have to deal with more viruses than before: I don't doubt that
> ripmime "got it wrong" sometimes on broken MIME, but whatever it was
> doing, it did allow clamd (for instance) to detect (for example) SirCam.
It will work with SirCam, but it may not work with others in the future.
> Would it not be feasible to consider, in addition to the demime_action, to
> have an option "attempt to unpack broken MIME" which would make a "best
> effort" attempt?
Yes, I agree on that. However, it does mean an increased overhead for me
in programming, since I'll have to do a lot more sanity checking. I may
add that in the future that that will take a while.
> Another thing: even if the "broken MIME catcher" has 0% false positives,
> the message it produces (e.g. "This message has broken MIME (base 64
> length is not a multiple of 4 characters") is very technical and not as
> intelligible as "This message contains a virus..." in the event that it
> lands back in an end users' mailbox.
I know. You can now easily define a "wrapper" text with
exiscan_demime_rejectmsg that can include a more verbose explanation of
the general problem.
Thanks a lot for your input. Are there other opinions on that topic out
there ?
cheers,
/tom
--
Tom Kistner <tom@???>
ICQ 1501527 dcanthrax@efnet
http://duncanthrax.net
