Author: Dean Brooks Date: To: exim-users Subject: Re: [exim] queue runner question
On Tue, Sep 20, 2011 at 12:08:29PM -0400, Derek Knapp wrote: > Is there any kind of option for the minimum number of queue runners?
>
> I have a shell script that adds emails to the queue (using
> /usr/sbin/exim -bS -odq) but every time I check the server there are
> thousands of emails in the queue waiting to be sent.
>
> I have seen many posts online about using queue_only option, but non
> really say how they handle starting up the queue runners.. if I
> change my shell script to remove the -odg, the emails are sent out
> much faster than 1st queuing them, but I figured adding them to the
> exim queue 1st would allow exim to send them in batches (for example
> if there are 100 emails going to hotmail, it could send them all at
> once)
>
> Is there a proper way to handle the starting of queue runners?
> should I just create an init script that runs exim -q1m 100 times
> every time the server starts?
When an Exim queue runner process starts, it begins working through
the queue linearly, either in random order or based on message
received order, depending on if you've set the "queue_run_in_order"
option.
The problem is that if a queue runner hits a message to be delivered
remotely, and the remote server isn't responding quickly, it may take
up to 5 minutes for it to time-out, defer the delivery, and then move
on to the next message. If you have a lot of messages on the queue,
and a lot of remote hosts that are slow to respond, it could literally
take hours or even days for a single queue runner to move through the
queue.
To offset this, you set "queue_run_max" to the maximum number of
simultaneous queue runners you want to ever have running at once.
This value doesn't start up queue runners, or guarantee that many will
be running. It's just a limit on the maximum allowed. Exim locks
messages during delivery so that two queue runners dont process the
same message at once. If you have powerful enough hardware, you can
set queue_run_max to be fairly high.
Queue runners are started automatically, one at a time, every so many
minutes as specified in the "-qXXm" command line option. If you have
it set to something like "-q15m", a new queue runner will only start
once every 15 minutes. If you are queueing all your email, this may
be way too high a value. If your hardware is powerful enough, and you
want the queue runners to ramp up quickly, you could set it to
something even as low as "-q30s" or "-q20s". Be careful though! It
all depends on how quickly you want queue runners to ramp up to the
maximum and what your hardware is capable of.
Couple of other things, in no particular order:
1. If you restart Exim with a sighup restart signal, the existing
queue runners keep running, but Exim resets it's internal queue count
to zero. That means you'll have twice the number of maximum queue
runners running at some point. That can quickly kill a server if you
send a restart signal several times in a row. I've gotten in the
habit of killing Exim completely to avoid that.
2. If you frequently have a large queue due to slow-responding hosts,
I would recommend creating a separate instance of Exim, perhaps on a
different virtual IP address or port number, and have a router in your
main configuration that forwards messages over to this "fallback"
instance of Exim if a message can't be delivered quickly. This keeps
your main queue lean so that properly-responding mail hosts get
deliveries fast and slower-responding mail hosts get put into a
cess-pool, so to speak. If you need example routers to do this, let
me know.
3. My personal preference is to not automatically queue everything,
but to set the timeout values on my main remote transport very low. I
have the smtp connection timeouts and command timeouts set to 40
seconds (40 works well because many hosts delay up to 30 seconds
before responding). If a remote server responds quickly, the message
gets delivered immediately. Otherwise, it goes to the fallback
instance of Exim. This makes for very fast deliveries to 95% of the
mail servers, and then a separate queue for everyone else. Why make
95% of deliveries suffer because of the 5% of ill-behaving hosts?
