Re: [exim] large queues

On Fri, Jul 28, 2006 at 09:54:51AM +0100, Philip Hazel wrote:
> On Fri, 28 Jul 2006, Peter D. Gray wrote:
>
> > I am about to implement an exim configuration whereby mail
> > tagged as spam by our gateways (not exim) is
> > frozen in the queues for later delivery. I intend to
> > deliver it at night. I use the queue_only option to
> > put all mail into the queue in any case, and I have
> > qrunners which perform actual delivery. The spam
> > mail will be frozen by a global filter.
> >
> > Obviously, the size of the mail queue is about to get
> > real large. I anticipate having 20K to 100K messages
> > in the queue. I am running the split_queue_directory option.
>
> Exim was never designed to operate efficiently with large queues. The
> whole philosophy was to move mail quickly without hanging on to it. In
> our environment, fewer than 5% of messages have to be retried. (See
> chapter 3 of my book. Or you should have come on my course last week.
> Oh, I see you are in .au; a bit far. :-) Of course, 5% of our volume is
> nowadays a lot more messages than it was 11 years ago when I first
> thought of Exim.
>

Yes, and spam is completely out of control now and I see everyone
desperately searching for solutions no matter how bizarre.
And this is leading to email systems being used in ways they
were never intended. I do understand the problem.


> If you do as you suggest, you are going to be mixing two different
> kinds of message in the same queue: those you are holding on to and
> those that are having delivery problems. This makes the administration
> harder.
>

Hmmmmmmmm.... would it be possible to simply rename the -H files
to -F when a message is frozen. That way it would be fast
to ignore frozen messages without the overhead of an open
and read.

> > It would be nice if exim understood anternate queues, that
> > way I could use the filter action to change the mail
> > queue and freeze at the same time (RFE please).
>
> There is a Wish List item about his, but as it is a major re-design, it
> ain't going to happen quickly, if at all.
>

The option of using the file names to reflect message state might
be an easier change and might offer significant performance gains
when processing queues.

>
> I suggest you treat this in a similar way to storing mail for dial-up
> hosts. The advice I give for this is to deliver such messages in BSMTP
> format into files (per user, per domain, or whatever). In the dial-up
> situation, this allows you to limit the volume, time out individual
> messages, etc.
>
> Then shovel them back into Exim using -bS when you want them delivered,
> with an appropriate indication (e.g. protocol setting) so they don't end
> up back in files again.
>

I will think about this, thanks.

Regards,
pdg

--

See mail headers for contact information.