hi all
i run a machine which is a high load email processor
essentially it receives emails from SMTP, parses them with procmail, manipulates them with php+mysql (db is on a separate machine) and then emails up to 6 other recipients with the parsed data. all the mails are collected using pop3.
the system has become quite well utilised over the last few months, and i am experiencing some SERIOUS performance problems. i'm running freebsd(3.5) and exim(3.03). my load averages on the machine are next to nothing, i have 512MB RAM and 2 GIG of swap space. during the day when mails are being processed, i can barely do anything on the box. people are waiting up to 1 minute for socket connections. if i run any command on the box, even something basic like ls or a grep, it takes FOREVER to execute. if i take a look at my file utilisation with pstat, i have over 600 files in use. there are about 1300 users on the machine. i don't get that many smtp connections, but when i do - they send a LARGE amount of mails to me. so i don't believe the performance problem is network related, i believe its filesystem related - even though it is a SCSI drive - it may be because of all the file locks and concurrent file usage.
from reading around today, i believe my problem may be the high impact usage of the /var/mail/username mboxes. if i look at my exim logs and my cucipop logs, i see a LARGE amount of spool file locks, errors opening mail boxes. often cucipop is unable to delete mails from the mbox which results in recurring mail problems during POP3 pickup.
i've tried many things, now i believe the way forward is to change my configuration from mbox to maildir style. i haven't been able to find that much obvious documentation on this (maybe i'm looking in the wrong places!). so i've got a couple of questions to ask if you don't mind...
my understanding of the difference between mbox and maildir is that maildir uses a separate file for each message and it has three directories it uses to process mail. at the moment i use /var/mail/user as the location for mail delivery. if i was to change to delivery mechanism - does this directory configuration stay the same which will leave me with /var/mail/user/tmp and so on for the mailbox? if so - how, exactly, do i change my configure (attached) to implement maildir.
secondly. and this is more of a concern to me. this system is used to manipulate financial data. i cannot be responsible for losing mails. i've been trying to find out if there are any pop3 servers that can SIMULTANEOUSLY support mbox and maildir for pickups - but haven't had much joy. if there is such a daemon, please tell me what it is.
if not, i've seen stuff around newsgroups and so on about qmail migration tools that will move mbox->mailbox. is there such a thing for exim?
help in this regards would be VERY MUCH appreciated. i'm having sleepless nights about the performance of this damn thing!
cheers
mike
auto_thaw = 30m
#check_dns_names = true
check_log_space = 10M
check_spool_space = 10M
delay_warning = 4h:8h:24h
rfc1413_query_timeout = 0s
smtp_accept_queue_per_connection = 500
#deliver_load_max = 20.0
#deliver_queue_load_max = 20.0
dns_again_means_nonexist = false
dns_retrans = 10m
dns_retry = 5
errors_address = "postmaster"
freeze_tell_mailmaster = true
headers_check_syntax = true
headers_checks_fail = false
headers_sender_verify = true
headers_sender_verify_errmsg = true
helo_strict_syntax = true
helo_verify = false
#host_lookup_nets = 0.0.0.0/0
ignore_errmsg_errors = true
primary_hostname = messenger.co.za
log_level = 6
qualify_domain = messenger.co.za
local_domains = messenger.co.za
local_domains_include_host_literals
exim_user = root
exim_group = mail
never_users = root : root : daemon : bin
relay_domains = messenger.co.za
sender_verify
pid_file_path = /var/run/exim%s.pid
end
######################################################################
# TRANSPORTS CONFIGURATION #
######################################################################
# ORDER DOES NOT MATTER #
# Only one appropriate transport is called for each delivery. #
######################################################################
# A transport is used only when referenced from a director or a router that
# successfully handles an address.
# This transport is used for delivering messages over SMTP connections.
remote_smtp:
driver = smtp
delay_after_cutoff
# This transport is used for local delivery to user mailboxes. By default
# it will be run under the uid and gid of the local user, and requires
# the sticky bit to be set on the /var/mail directory. Some systems use
# the alternative approach of running mail deliveries under a particular
# group instead of using the sticky bit. The commented options below show
# how this can be done.
local_delivery:
driver = appendfile
file = /var/mail/${local_part}
delivery_date_add
envelope_to_add
return_path_add
group = mail
mode = 0660
# This transport is used for handling pipe deliveries generated by alias
# or .forward files. If the pipe generates any standard output, it is returned
# to the sender of the message as a delivery error. Set return_fail_output
# instead of return_output if you want this to happen only when the pipe fails
# to complete normally. You can set different transports for aliases and
# forwards if you want to - see the references to address_pipe below.
address_pipe:
driver = pipe
return_output
# This transport is used for handling deliveries directly to files that are
# generated by aliassing or forwarding.
address_file:
driver = appendfile
delivery_date_add
envelope_to_add
return_path_add
# This transport is used for handling autoreplies generated by the filtering
# option of the forwardfile director.
address_reply:
driver = autoreply
end
######################################################################
# DIRECTORS CONFIGURATION #
# Specifies how local addresses are handled #
######################################################################
# ORDER DOES MATTER #
# A local address is passed to each in turn until it is accepted. #
######################################################################
# Local addresses are those with a domain that matches some item in the
# "local_domains" setting above, or those which are passed back from the
# routers because of a "self=local" setting (not used in this configuration).
# This director handles aliasing using a traditional /etc/aliases file.
# If any of your aliases expand to pipes or files, you will need to set
# up a user and a group for these deliveries to run under. You can do
# this by uncommenting the "user" option below (changing the user name
# as appropriate) and adding a "group" option if necessary. Alternatively, you
# can specify "user" on the transports that are used. Note that those
# listed below are the same as are used for .forward files; you might want
# to set up different ones for pipe and file deliveries from aliases.
system_aliases:
driver = aliasfile
file = /etc/aliases
search_type = lsearch
file_transport = address_file
pipe_transport = address_pipe
user = root
# This director handles forwarding using traditional .forward files.
# If you want it also to allow mail filtering when a forward file
# starts with the string "# Exim filter", uncomment the "filter" option.
# The no_verify setting means that this director will be skipped when
# verifying addresses if sender_verify or receiver_verify is set (though
# they are not set by default). Similarly, no_expn means that this director
# will be skipped if smtp_expn_hosts is set to allow any hosts to use the
# EXPN command.
# The check_ancestor option means that if the forward file generates an
# address that is an ancestor of the current one, the current one gets
# passed on instead. This covers the case where A is aliased to B and B
# has a .forward file pointing to A.
# The three transports specified at the end are those that are used when
# forwarding generates a direct delivery to a file, or to a pipe, or sets
# up an auto-reply, respectively.
userforward:
driver = forwardfile
file = .forward
no_verify
no_expn
check_ancestor
# filter
file_transport = address_file
pipe_transport = address_pipe
reply_transport = address_reply
# This director matches local user mailboxes.
localuser:
driver = localuser
transport = local_delivery
end
######################################################################
# ROUTERS CONFIGURATION #
# Specifies how remote addresses are handled #
######################################################################
# ORDER DOES MATTER #
# A remote address is passed to each in turn until it is accepted. #
######################################################################
# Remote addresses are those with a domain that does not match any item
# in the "local_domains" setting above.
# This router routes to remote hosts over SMTP using a DNS lookup with
# default options.
lookuphost:
driver = lookuphost
transport = remote_smtp
# This router routes to remote hosts over SMTP by explicit IP address,
# given as a "domain literal" in the form [nnn.nnn.nnn.nnn]. The RFCs
# require this facility, which is why it is enabled by default in Exim.
# If you want to lock it out, set forbid_domain_literals in the main
# configuration section above.
literal:
driver = ipliteral
transport = remote_smtp
end
######################################################################
# RETRY CONFIGURATION #
######################################################################
# This single retry rule applies to all domains and all errors. It specifies
# retries every 15 minutes for 2 hours, then increasing retry intervals,
# starting at 1 hour and increasing each time by a factor of 1.5, up to 16
# hours, then retries every 8 hours until 4 days have passed since the first
# failed delivery.
# Domain Error Retries
# ------ ----- -------
* * F,2h,15m; G,16h,1h,1.5; F,4d,8h
end
######################################################################
# REWRITE CONFIGURATION #
######################################################################
# There are no rewriting specifications in this default configuration file.
# End of Exim configuration file
