Good morning!
I'm hoping someone can give me a mental whack on the side of the head to see
what I'm overlooking in this situation with some Exim problems. The
answer is going to be painfully obvious with 20-20 hindsight, but I've been
working on this for three days, searching Google, poring over manuals, and
it's just not popping out at me.
Configuration is this:
* Two inbound mail routing servers receive SMTP connections from external
hosts but never from internal hosts. These servers fan the mail out to
five downstream mailbox servers. The routing servers do not do virus or
spam scanning or any other CPU-intensive processes; they just determine
the recipient's physical mailbox host and deliver the raw message. All
outbound traffic, including bounce messages, is fed to two smarthost
outbound routers, which are not part of this discussion because they are
easily keeping up with their load.
* Exim 4.2 and OpenLDAP 2.0.27 running on a 2.4GHz uniprocessor box, with
2.5 GB of RAM and fast SCSI disk drives. Filesystems are ext3. Exim is set
up to use split queue directories (62 of them) so that you don't have a
huge number of files in a single directory. System is RH Linux 8.
* LAN is all 100 megabit Ethernet, dedicated switched segments. WAN is DS3,
and they have checked utilization and found it to be low (<50%).
* All mail is being correctly delivered as far as we are aware. No mail loops
evident, no abundance of errors in logs, or anything like that.
* Volume of mail is about 20K messages/hour. No virus or spam scanning happens
on the mailrouter boxen, of which there are two (so each should be handling
10K messages/hour). Spray-out of connections is via round-robin DNS.
* Exim is configured to do no reverse DNS lookups and no IDENTD lookups. The
route to the downstream mailbox servers is done with an "accept" router
and an explicit hostname in the associated transport. The transport also has
the gethostbyname option TRUE to allow the /etc/hosts file to be used to
look up the IP number for the remote.
* Internet commentaries on Exim suggest that each of these two servers ought
to be able to handle several times the load we are putting on them.
* The routers in question handle only inbound traffic, and spray it out to
one of five mailbox servers depending on the user record in LDAP. I have
verified that LDAP's performance is fast enough not to be a factor, and in
in any case the problem is happening at a point in the SMTP protocol *before*
the LDAP lookup is involved.
* Performance parameters look like this right now, though I've tried other
settings (many combinations):
# host_lookup = * [default is unset]
rfc1413_hosts = *
rfc1413_query_timeout = 0s [suppress IDENT lookups]
queue_run_max = 60
# queue_only_load = 20
deliver_queue_load_max = 20
smtp_accept_queue_per_connection = 400
smtp_accept_max_per_host = 2
smtp_accept_max_per_connection = 800
smtp_accept_max = 300
recipients_max = 18
helo_allow_chars = _
split_spool_directory = yes
* The downstream mailbox servers have empty queues and show very light loading.
They are set up with wide inbound MTA pipes, but the routers with the
performance problems seem not to use those pipes effectively.
* I have tried tuning kernel sysctl parameters as follows:
tcp_max_syn_backlog 1024 (default), 512, and 2000
tcp_rfc1337 0 (disabled, default), 1 (enabled)
The RFC1337 made a huge difference in performance on the downstream mailobox
servers, but seems to have little effect on the upstream inbound routers.
* LDAP is set up to allow up to 750 simultaneous threads in slapd. The most Exim
instances we've ever had running is about 600, but 300-400 is more typical.
Now, to the problem symptoms:
* vmstat and uptime, monitored for up to 30 hours, consistently show CPU idle
of about 85-90%.
* Ethernet traffic is on the order of 200Kbytes/second *total*. This seems
extraordinarily low to me.
* If you use telnet to connect to port 25 on the downstream mailbox servers,
you get the Exim banner instantaneously.
* If you use telnet to connect to the inbound routers, however, you experience
a delay of 10 to 45 seconds before getting the banner. THIS IS THE BIGGEST
PART OF THE PROBLEM, in my opinion, and I think this is what's delaying
the inbound messages.
* Memory usage is consistently moderate, with Linux having about 900MB of RAM
left over for disk cache.
* netstat shows tons of connections in various states. Most of the connections
to the LDAP server seem to be in TIME_WAIT status most of the time. Many of
the inbound SMTP connections are in SYN_RECV status.
* LDAP response times, even when Exim is at its slowest, are sub-second for
queries that I've carefully selected so that I know the results are *not*
cached in RAM. For users who have been recently queried (that is, are in
RAM cache in slapd), the response is instantaneous. I'm convinced that LDAP
is not the source of the delay.
I'm utterly at loss to explain why this system performs so slowly, when there
are so many resources left on the system. My instinct says the problem is
that incoming connections and outgoing connections can't get sockets, but I
don't know of any other things to change to fix it. Or else there are reverse
DNS lookups happening in Exim that I'm not aware of.
Any suggestions would be much appreciated.
Scott
--
-----------------------+------------------------------------------------------
Scott Courtney | "I don't mind Microsoft making money. I mind them
courtney@??? | having a bad operating system." -- Linus Torvalds
http://www.4th.com/ | ("The Rebel Code," NY Times, 21 February 1999)
