Re: [exim] Exim 5.x

Phil Pennock wrote:
> On 2011-05-19 at 12:35 +0100, Dominic Benson wrote:
>> On 19/05/11 11:50, Phil Pennock wrote:
>>>
>>> New queuing system refers to an approach to scale up the spool directory
>>> to something more queue-like, with segregated admin-defined queues (eg,
>>> "big_freemail_provider_x"). This is because while Exim is excellent at
>>> inbound mail, it doesn't always scale as well as some would like for
>>> outbound mail which can't be immediately delivered. Nothing has been
>>> done on this. Patches welcome.
>>>
>> Is this AKA bug 336? It sounds quite interesting, so I think I might
>> have a look at making some inroads into the problem.
>>
>> If there are are any notes/thoughts about behaviour/config/use it would
>> be handy to hear them.
>
> I wasn't aware of 336, but yes it's related.
>
> At present, there's split_spool_directory, which divides things up with
> one level of hashing, and then some people script their own queue-runner
> launchers, running in parallel over sub-trees of the split spool instead
> of having the Exim daemon launch runners over everything, which compete
> with each other.
>
> Nothing more specific was discussed, that I either recall or find in the
> minutes; we all understood the general problem.

Known bound, yes.

Problem?

Not sure.

I stand on the position that WHEN one is loading Exim so heavily that it
HITS that sort of bound, one has far too many eggs in one basket
downtime-risk-wise, and should split that load over multiple Exim (free)....

... on multiple separate boxen (generally cheap). Or at least usually
'cheapER' than answering complaints from such a huge user base - or
rushing to a remote data center.

>
> If we use a new sub-directory of spool_directory to hold named queues,
> then previous Exim installs won't know of the content, but it should be
> fairly easy to script a rollback tool which recombines queues into the
> original queue.
>
> Thinking briefly (while tired and prone to mistakes):
> You'd need a way to declare "move this message to this other queue",

.... or just a small critter managing ephemeral softlinks, perhaps?

> perhaps a way to restrict Routers to only apply to messages in certain
> queues (seems optional as a performance optimisation, *could* be useful
> for some more sophisticated Router rule bifurcations) and either a way
> to control counts of queue-runners per named queue or an API to do so.

Probably do-able already with a bit of metal bending.

For sure routers have no problem delivering wherever admin-config
directed, and retrieval is just the other side of the same coin should
it be seen useful.

> Probably best to have a global config option defining a list of allowed
> queue names, so that a typo doesn't end up putting 100k messages into
> "gmial" where they sit idle. ;)
>

Worse has happened.

Left a delimiter off the front of a complex SQL call once. Messages
disappeared for a while.

Took 'ls' to the mailstore.

Found a new user with the path to Maildir not the *result* of the SQL
returned from the DB...

.. but rather the full-text of the unbounded SQL call *itself* ..as the
directory name.

:-(

> Beyond that, I think there's a lot of freedom for the implementor to
> make their own decisions. :) Have at it!
>
>

Aye. 'Enough rope ....'

Bill