Re: [exim] SMTP processing hangs during bayes sync

On Thu, 31 Aug 2006, W B Hacker wrote:

> Bodo Gelbe wrote:
>
>> I've the following problem with exim and exiscan:
>>
>> Exim calls spamd in the DATA ACL (for messages from
>> external hosts).
>>
>> spamassassin/spamd is configured with
>>
>> bayes_auto_learn 1
>> bayes_journal_max_size 0
>> bayes_auto_expire 0
>> bayes_learn_to_journal 1
>>
>>> From time to time (cronjob) the database and the journal are
>> synchronized (sa-learn --sync). During the synchronization
>> process SMTP requests are hanging for 20 to 50 seconds (even
>> those from local hosts, which will not be scanned by spamd).
>> Number of exim processes are increasing.
>
> Does 'exiwhat' confirm that local traffic is in the pile of slowed-down
> processes? Does it tell you anyhting else that might be useful?

shows an increasing number of processes with status
"handling incoming connection from ..."

>
> What is CPU/memory load at the time?

Low.

>
>>
>> Looks like something is locked within exim before spamd
>> is called, which locks out processing for messages not
>> undergoing spamd processing.
>>
>
> May be that those that *are* awaiting SA have used up enough resources, limits,
> that the others are simply awaiting earlier processes to finish?

I've tried a "sendmail -d+all ..." during bayes sync and it shows
two delays:

08:32:05 3749 Date: Fri, 01 Sep 2006 08:32:05 +0200
08:32:05 3749
08:32:13 3749 Data file written for message 1GJ2ZR-0000yT-OK


08:32:13 3749 Writing spool header file
08:32:40 3749 Size of headers = 278


seems something in the spool area is locked before the call to
exiscan and exiscan is waiting for the lock of the bayes databases
to be released -> processing of the incoming message stucks.

>
> Are your various limits all at default, or?
>
>> Any idea how to avoid the delay (dont't want a transport
>> because we're doing greylisting based upon the spam level)?
>>
>> kr, Bodo
>>
>
> Changing the scheduled sync to low-load time-of-day, or providing greater
> resources aside, I'll suggest a contrarian approach:
>
> - Check and see how many times has the Bayes point contribution to the total SA
> score been sufficiently high to 'tip the balance' when a given message was not
> *already* firmly tagged as Spam by other rules, OR deniable for protocol,
> invalid recipient/domain, format, attachment, MIME, viral reasons?
>
> Hint: In the as-issued SA, most Bayes scores are so miniscule they rarely
> contribute enough to matter, yet Bayes consumes significant resources.
>
> Worse - Bayes on a server where user traffic can be so very different, one from
> another, seems to get confused far more easily than when run on an individual
> user's MUA. (pays its way on Mozilla/Thunderbrd and others...)
>
> Server-side, we've shut Bayes off entirely with more beneficial results than not.
>
> Greylisting, BTW, is likely to more than double your connection load, (retry may
> be idioticlly rapid for zombies) - spawning child processes that may not go all
> the way through to the DATA phase, but will certainly consume resources.


Yes but we're using greylisting only for selected mailboxes.


>
> Might help to run 'queue_only', adjust your queue runner interval 'til the
> resources balance better.

Some users are complaining because of the pause during the SMTP session.
They're using pine with SMTP support (not sendmail).

It's not a resource problem. From my point of view it's a
problem with setting/releasing locks.


>
> YMMV,
>
> Bill

kr, Bodo