hi folks,
bit of a long story, i've been using exim4 since around... 2002 i
think - a long time. i used to use it with cyrus 2.2. i've been
through a number of different types of configurations, but the one i'm
primarily using it for is to do forwarding-only to a wide range of
different email servers, and i use it for sending on email out (with a
secure password, and limited only to my VPN).
most of the email is forwarded to gmail. the issue i've got is that
occasionally, rather than be forwarded, messages are detected as SPAM,
and gmail very kindly says "hmmm, you seem to be sending rather a lot
of SPAM, we're going to rate-limit you" and they send i believe it's a
504 error code (not sure it's exactly that).
as a result the message ends up in a queue in exim4 - which is
*precisely* what i do not want to be happening - and it gets retried,
retried, retried and retried - end result is that it completely
trashes the delivery time of all other messages because gmail's
servers *remain* in the "this server's sending spam let's keep on
rate-limiting them" mode.
the ideal scenario is quite clear and quite obvious: under no
circumstances do *any* storing, under no circumstances *ever* queue a
message: the only thing that should ever happen is: deliver at *MTA*
time. do not even reply back to the client that's connected to us
until the ongoing server has responded. and, if there is ever a
"reject" or other type of message, simply pass that on as well.
now, many people would say "why are you using exim4??? surely you
should just drop it and use postfix or ssmtp???" answer: because i
bloody well don't want to: i *like* exim4, thank you :) but also,
because i do multiple domains, and because i have that
password-protected way to send outgoing emails from my server. so
it's *not* just a store-and-forwarder.
if someone could have a look and suggest some modifications - bear in
mind i usually do cut/paste but do it quite well - i'd be very
grateful to hear how it is possible to completely stop queueing and
storing... *without* disrupting all the other parts of the
configuration. of particular note is that i use the "vdom_aliases"
setup which someone wrote. this server DOES NOT accept even ONE
SINGLE message in the "standard" way of receiving email with exim4.
this is THOROUGHLY DELIBERATE after years of experience showed that
exim4 really does not cope properly with delivery to local recipients
without also being used as a SPAM deliverer [i did say it was a long
story].
anyway the config is here:
http://lkcl.net/exim4.config - most of it
is debian-based, some of it isn't.
on a separate but related note: do any of the developers remember that
"tired old debian cyrus lmtp imap exim4 setup which was totally faulty
for years, with none of the debian developers doing anything about it,
and was totally and absolutely 100% the exclusive fault of the debian
developers, and was in absolutely no way a bug in exim4"? after
investigating this bug for a number of years, i did finally track down
what the problem was, and, nine years ago, reported it. however,
because the reaction of the exim4 developers of the time was so
vehement and absolutely adamant that it *had* to be a bug in the way
that debian recommended to set up cyrus lmtp delivery to exim4, there
was absolutely nothing that i could do to convince them otherwise, so,
simplest thing - stop communicating.
i'd be interested to hear if the bug was ever independently found (and
fixed) in the intervening time. basically, the bug was related to a
discrepancy - a difference - between the SMTP delivery error handling
and the LMTP delivery error handling, when a user does not exist, in
particular when a message is cc'd to dozens of recipients [several
being legitimate, several being fake].
the key to finding this bug was to have one exim4 server do forwarding
(actually at the client's site there were 5 such front-end servers,
set up as secure bastions), and another exim4 server do LMTP delivery
(into a cyrus imap server, or, actually there were 2 of these).
messages would be delivered en-masse down a single SMTP session from
an incoming (arbitrary) internet server, then forwarded by the
front-end exim4 servers to the "internal" exim4 servers. those
"internal" servers would then do en-masse LMTP delivery down a single
socket to the cyrus imap4 servers.
the problems came when someone on the internet - usually a spammer -
added in an illegitimate (completely fake) email address in amongst
several *legitimate* messages that were perfectly legal and valid
recipients. the front-end exim4 server would accept them all (because
it's not its job to verify the messages: it doesn't have acess to the
email identities - and here's the really important bit - FOR SECURITY
REASONS IT WAS ***NEVER*** GOING TO BE ALLOWED ACCESS TO THE LIST OF
EMAIL RECIPIENTS). those front-end servers *critically* relied on the
back-end exim4 servers to tell them "hey guys, identity not known,
please report that back to the sender ok?"
now, it turns out that the error-code that the backend exim4 servers
returned when attempting to deliver a (fake) message to a non-existent
recipient over LMTP was - and may still be - *DIFFERENT* from the
*NORMAL* error-code used when exim4 does SMTP forwarding.
but that's not all: when this not-correct-error-code is returned,
exim4 would treat *ALL* recipients differently, and would do the
following:
(1) store the message for retry (when it should definitely NOT have
done so, especially as it's SPAM and has both a fake "From" *and" a
fake "To" address).
(2) due to SMTP "verification" that had been enabled, the exim4 server
would try to contact someone who had, clearly, never sent the message
(because the "From" had been faked up), and try to deliver them a
"sorry, your message could not be delivered" message.
(3) once the retry timeouts had been reached, the exim4 server would
try to send out "sorry your message could not be delivered"
messages... to *faked-up senders". this is where it really, really
started to go pear-shaped, resulting in one instance a DNSBL entry and
in another instance someone reporting the server to the hosting
company.
all of this was NOT down to the "error of the sysadmin not checking
the recipient email addresses because that's what you're supposed to
do before trying to send message on to cyrus over LMTP" - that's just
silly. the error is down to the WRONG reaction of exim4 to an error
code when it is trying to *deliver* over LMTP.
to make this clearer: let's imagine that i had created a version of
ssmtp which could handle LMTP instead. so, let's imagine that we have
a simple and dumb SMTP server which takes incoming email and tries to
send it down an LMTP socket. no storing, no forwarding: error
messages are blindly reported back, email is blindly sent down the
socket.
what i am saying is: if, all those years ago, i had set up exim4 to
deliver to *that* instead of directly to cyrus via lmtp, THE PROBLEM
WOULD HAVE MIRACULOUSLY CURED ITSELF.
one of the problems that i had in trying to get across that this is an
actual and important bug in exim4 was that it was described to me that
it was "entirely my fault for copying the default debian instructions,
and not doing verification of the email addresses prior to putting
them down the LMTP pipe". that however is just an excuse: it doesn't
get the problem fixed, and there are actual genuine reasons why you do
not - and should not - allow a front-end server to act in any way
other than as a "blind dumb forwarder". there are certain
circumstances where you *have* to rely on an SMTP server reacting to
(or giving) the correct error codes.
so, is that now clearer, as to what the problem was (is?) - and has it
now been fixed? if so, it would be great, because i can stop using
gmail and i can have another go at installing cyrus.
many thanks,
l.
---
crowd-funded eco-conscious hardware:
https://www.crowdsupply.com/eoma68
