Re: [exim] Exim with SQL message store

Marten Lehmann wrote:
> Hi,
>
>> I'd go for the other option: keep the existing logging, and *also* copy
>> it into a database, and then query that database. This could be
>> real-time (by extending your syslog) or near-real-time (by having
>> log-file watchers), or batched in cron every n minutes ...
>
> I was not asking for logging. SQL-logging might be nice as well, but it
> is more important, that all queue relevent need to be stored in a sql
> database. This would improve performance a lot. Right now exim has to
> parse thousands of files for find out which is the next one for a
> delivery attempt. A sql database which a fixed column set would be much
> simpler and the real message wouldn't have to be stored as blob, it
> could still be stored as a textfile.
>
> Wouldn't this be a goal for exim in general to improve the meta-data in
> queue data in a structured storage like SQLite?
>
> Regards
> Marten
>

If exim routinely has to 'parse thousands of files' in queue you may have your
system set up in a highly sub-optimal manner. Hundreds is a more realistic top
count, even on a busy server, and dozens not out of reach. A lightly-loaded
server might seldom see TWO in the queue for more than a minute or so.

My settings are the default to fire a queue-runner for every processed
connection - eg NOT setting to 'queue_only'. That should be all you need.

I also trigger a clean-up runner every 55 seconds - primarily for non-smtp
traffic, but that is more aggressive than generally needed. Far more.

Exim's file-system-based queue is not the fastest on Earth, but putting actual
traffic into an(y) SQL DB is bound to be slower yet. Not only is SQL more
'overhead' - but it ultimately has to also use the file system for storage anyway.

TANSTAAFL

Putting the queue entirely in fast RAM is quick - but carries a serious risk of
loss unless that is expensive BATRAM AND the OS AND MTA can actually recover it
seamlessly AND NOT just clear it or flag it to be ignored on restart. Not at all
guaranteed w/o expert setup and testing.

Now - other reasons for 'thousands' are that you are encountering *serious*
delivery failure/retry load.

That bears investigation - more especially so if you are running a MLM and/or
otherwise trying to transfer dodgy traffic.

Not to forget - message-quality quite aside - if your own server's credentials
are flawed in any way, you might be getting indefinite grey-listing or 'defer'
instead of deny from clever far-end boxen.

Best you dig into the 'why' of such a queue load. You may not need any change to
queuing at all.

Proof? Very, very few of the rest of us have had such a need, and there are
among us ISP's and universities with *massive* traffic loads on Exim.

HTH,

Bill Hacker