Re: [exim] Issues with headers_remove in router ::

Top Page

Reply to this message
Author: Shamim Shahriar
Date:  
To: exim-users
Subject: Re: [exim] Issues with headers_remove in router ::
Hello All,

I think I found something that may be the reason for the headers_remove to
fail (also confirmed by Valki at the exim4u list)

Inside headers_remove, the lookup mysql takes only upto a certain length
(too small, in my opinion), and discards anything beyond that length --
therefore generating a "missing }" error.


setup: FreeBSD 9 Stable
Exim version : 4.84
Vexim latest, with local modification that does not interfere with Exim
itself

snippets:
15:45:32 53510 failed to expand: select users.sa_tag * 10 from
users,domains where local
15:54:32 54370 failed to expand: select users.sa_tag * 10 from
users,domains where local
15:59:39 55092 failed to expand: select users.sa_tag * 10 from
users,domains where local
16:04:55 55934 failed to expand: select users.sa_tag * 10 from
users,domains where `loca
16:16:03 58090 failed to expand: select users.sa_tag * 10 from
users,domains }{$value}fa

Please note the last line, where it stopped at the same length.


Test 01 :: double colons
-------------------------

headers_remove = ${if or { { <{$spam_score_int}{1} } \
        { <{$spam_score_int}{${lookup mysql{select users.sa_tag * 10 from
users,domains \
        where localpart = '${quote_mysql::$local_part}' \
        and domain = '${quote_mysql::$domain}' \
        and users.on_spamassassin = 1 \
        and users.domain_id=domains.domain_id}{$value}fail}} } \
        { eq {0}{${lookup mysql{select users.sa_tag * 10 from users,domains
\
        where localpart = '${quote_mysql::$local_part}' \
        and domain = '${quote_mysql::$domain}' \
        and users.on_spamassassin = 0 \
        and users.domain_id=domains.domain_id}{$value}fail}}} \
        } {X-Spam-Score::X-Spam-Report} }




Result:
-------
15:45:32 53510 expanding: ${if >{$spam_score_int}{${lookup mysql{select
users.sa_tag * 10 from users,domains where localpart =
'${quote_mysql:$local_part}' and domain = '${quote_mysql:$domain}' and
users.on_spamassassin = '1' and users.domain_id=domains.domain_id
}{$value}fail}} {X-Spam-Flag: YES\n}{} }
15:45:32 53510    result: X-Spam-Flag: YES
15:45:32 53510
15:45:32 53510 expanding: $spam_score_int
15:45:32 53510    result: 27
15:45:32 53510 expanding: 1
15:45:32 53510    result: 1
15:45:32 53510 expanding: $spam_score_int
15:45:32 53510    result: 27
15:45:32 53510 failed to expand: select users.sa_tag * 10 from
users,domains where local
15:45:32 53510    error message: missing } at end of string
15:45:32 53510 failed to expand: ${lookup mysql{select users.sa_tag * 10
from users,domains where local
15:45:32 53510    error message: missing } at end of string
15:45:32 53510 failed to expand: ${if or { { <{$spam_score_int}{1} } {
<{$spam_score_int}{${lookup mysql{select users.sa_tag * 10 from
users,domains where local
15:45:32 53510    error message: missing } at end of string inside
"or{...}" condition
15:45:32 53510 virtual_domains router: defer for
postmaster@???
15:45:32 53510   message: virtual_domains router failed to expand "${if or
{ { <{$spam_score_int}{1} } { <{$spam_score_int}{${lookup mysql{select
users.sa_tag * 10 from users,domai
ns where localpart = '${quote_mysql::$local_part}' and domain =
'${quote_mysql::$domain}' and users.on_spamassassin = 1 and
users.domain_id=domains.domain_id}{$value}fail}} } { eq {0}{${lookup
mysql{select users.sa_tag * 10 from users,domains where localpart =
'${quote_mysql::$local_part}' and domain = '${quote_mysql::$domain}' and
users.on_spamassassin = 0 and
users.domain_id=domains.domain_id}{$value}fail}}} }
{X-Spam-Score::X-Spam-Report} }": missing } at end of string inside
"or{...}" condition




============================
Test 02 :: remove mysql_quote
-----------------------------

headers_remove = ${if or { { <{$spam_score_int}{1} } \
        { <{$spam_score_int}{${lookup mysql{select users.sa_tag * 10 from
users,domains \
        where localpart = '{$local_part}' \
        and domain = '{$domain}' \
        and users.on_spamassassin = 1 \
        and users.domain_id=domains.domain_id}{$value}fail}} } \
        { eq {0}{${lookup mysql{select users.sa_tag * 10 from users,domains
\
        where localpart = '{$local_part}' \
        and domain = '{$domain}' \
        and users.on_spamassassin = 0 \
        and users.domain_id=domains.domain_id}{$value}fail}}} \
        } {X-Spam-Score::X-Spam-Report} }




Result:
-------
15:54:32 54370 expanding: $spam_score_int
15:54:32 54370    result: 27
15:54:32 54370 expanding: 1
15:54:32 54370    result: 1
15:54:32 54370 expanding: $spam_score_int
15:54:32 54370    result: 27
15:54:32 54370 failed to expand: select users.sa_tag * 10 from
users,domains where local
15:54:32 54370    error message: missing } at end of string
15:54:32 54370 failed to expand: ${lookup mysql{select users.sa_tag * 10
from users,domains where local
15:54:32 54370    error message: missing } at end of string
15:54:32 54370 failed to expand: ${if or { { <{$spam_score_int}{1} } {
<{$spam_score_int}{${lookup mysql{select users.sa_tag * 10 from
users,domains where local
15:54:32 54370    error message: missing } at end of string inside
"or{...}" condition
15:54:32 54370 virtual_domains router: defer for
postmaster@???
15:54:32 54370   message: virtual_domains router failed to expand "${if or
{ { <{$spam_score_int}{1} } { <{$spam_score_int}{${lookup mysql{select
users.sa_tag * 10 from users,domains where localpart = '{$local_part}' and
domain = '{$domain}' and users.on_spamassassin = 1 and
users.domain_id=domains.domain_id}{$value}fail}} } { eq {0}{${lookup
mysql{select users.sa_tag * 10 from users,domains where localpart =
'{$local_part}' and domain = '{$domain}' and users.on_spamassassin = 0 and
users.domain_id=domains.domain_id}{$value}fail}}} }
{X-Spam-Score::X-Spam-Report} }": missing } at end of string inside
"or{...}" condition




============================
Test 03 :: put the variables properly
-------------------------------------

headers_remove = ${if or { { <{$spam_score_int}{1} } \
        { <{$spam_score_int}{${lookup mysql{select users.sa_tag * 10 from
users,domains \
        where localpart = '${local_part}' \
        and domain = '${domain}' \
        and users.on_spamassassin = 1 \
        and users.domain_id=domains.domain_id}{$value}fail}} } \
        { eq {0}{${lookup mysql{select users.sa_tag * 10 from users,domains
\
        where localpart = '${local_part}' \
        and domain = '${domain}' \
        and users.on_spamassassin = 0 \
        and users.domain_id=domains.domain_id}{$value}fail}}} \
        } {X-Spam-Score::X-Spam-Report} }




Result:
-------

15:59:39 55092 expanding: $spam_score_int
15:59:39 55092    result: 27
15:59:39 55092 expanding: 1
15:59:39 55092    result: 1
15:59:39 55092 expanding: $spam_score_int
15:59:39 55092    result: 27
15:59:39 55092 failed to expand: select users.sa_tag * 10 from
users,domains where local
15:59:39 55092    error message: missing } at end of string
15:59:39 55092 failed to expand: ${lookup mysql{select users.sa_tag * 10
from users,domains where local
15:59:39 55092    error message: missing } at end of string
15:59:39 55092 failed to expand: ${if or { { <{$spam_score_int}{1} } {
<{$spam_score_int}{${lookup mysql{select users.sa_tag * 10 from
users,domains where local
15:59:39 55092    error message: missing } at end of string inside
"or{...}" condition
15:59:39 55092 virtual_domains router: defer for
postmaster@???
15:59:39 55092   message: virtual_domains router failed to expand "${if or
{ { <{$spam_score_int}{1} } { <{$spam_score_int}{${lookup mysql{select
users.sa_tag * 10 from users,domains where localpart = '${local_part}' and
domain = '${domain}' and users.on_spamassassin = 1 and
users.domain_id=domains.domain_id}{$value}fail}} } { eq {0}{${lookup
mysql{select users.sa_tag * 10 from users,domains where localpart =
'${local_part}' and domain = '${domain}' and users.on_spamassassin = 0 and
users.domain_id=domains.domain_id}{$value}fail}}} }
{X-Spam-Score::X-Spam-Report} }": missing } at end of string inside
"or{...}" condition




============================
Test 04 :: tick all the mysql query
-------------------------------------

headers_remove = ${if or { { <{$spam_score_int}{1} } \
        { <{$spam_score_int}{${lookup mysql{select users.sa_tag * 10 from
users,domains \
        where `localpart` = '${quote_mysql::$local_part}' \
        and `domain` = '${quote_mysql::$domain}' \
        and `users.on_spamassassin` = 1 \
        and `users.domain_id`='domains.domain_id'}{$value}fail}} } \
        { eq {0}{${lookup mysql{select users.sa_tag * 10 from users,domains
\
        where `localpart` = '${quote_mysql::$local_part}' \
        and `domain` = '${quote_mysql::$domain}' \
        and `users.on_spamassassin` = 0 \
        and `users.domain_id`='domains.domain_id'}{$value}fail}}} \
        } {X-Spam-Score::X-Spam-Report} }



Result:
-------
16:04:55 55934 expanding: $spam_score_int
16:04:55 55934    result: 27
16:04:55 55934 expanding: 1
16:04:55 55934    result: 1
16:04:55 55934 expanding: $spam_score_int
16:04:55 55934    result: 27
16:04:55 55934 failed to expand: select users.sa_tag * 10 from
users,domains where `loca
16:04:55 55934    error message: missing } at end of string
16:04:55 55934 failed to expand: ${lookup mysql{select users.sa_tag * 10
from users,domains where `loca
16:04:55 55934    error message: missing } at end of string
16:04:55 55934 failed to expand: ${if or { { <{$spam_score_int}{1} } {
<{$spam_score_int}{${lookup mysql{select users.sa_tag * 10 from
users,domains where `loca
16:04:55 55934    error message: missing } at end of string inside
"or{...}" condition
16:04:55 55934 virtual_domains router: defer for
postmaster@???
16:04:55 55934   message: virtual_domains router failed to expand "${if or
{ { <{$spam_score_int}{1} } { <{$spam_score_int}{${lookup mysql{select
users.sa_tag * 10 from users,domains where `localpart` =
'${quote_mysql::$local_part}' and `domain` = '${quote_mysql::$domain}' and
`users.on_spamassassin` = 1 and
`users.domain_id`='domains.domain_id'}{$value}fail}} } { eq {0}{${lookup
mysql{select users.sa_tag * 10 from users,domains where `localpart` =
'${quote_mysql::$local_part}' and `domain` = '${quote_mysql::$domain}' and
`users.on_spamassassin` = 0 and
`users.domain_id`='domains.domain_id'}{$value}fail}}} }
{X-Spam-Score::X-Spam-Report} }": missing } at end of string inside
"or{...}" condition
16:04:55 55934 added retry item for R:postmaster@???:
errno=-1 more_errno=0 flags=0








============================
Test 05 :: change the content inside lookup mysql
-------------------------------------------------


headers_remove = ${if or { { <{$spam_score_int}{1} } \
                { <{$spam_score_int}{${lookup mysql{select users.sa_tag *
10 from users,domains \
                }{$value}fail}} } \
                { eq {0}{${lookup mysql{select users.sa_tag * 10 from
users,domains \
                }{$value}fail}} } \
                } {X-Spam-Score::X-Spam-Report} }



Result:
-------
16:16:03 58090 failed to expand: ${lookup mysql{select users.sa_tag * 10
from users,domains }{$value}fa


On 8 September 2014 12:46, Shamim Shahriar <shamim.shahriar@???>
wrote:

> Hi Jeremy,
>
> Just to check if my understanding of the problem is correct, I modified
> the headers_remove section (without adding the bracket you suggested) in
> the two following ways
>
> #=== Way 01 :: hard coded value ===
>   headers_remove = ${if or { { <{$spam_score_int}{1} } \
>     { <{$spam_score_int} {20}} \
>     { eq {0}{10}} \
>     } {X-Spam-Score::X-Spam-Report} }

>
> #================
>
> #=== way 02 :: empty value ===
>  headers_remove = ${if or { { <{$spam_score_int}{1} } \
>                { <{$spam_score_int}{${lookup mysql{}{$value}fail}} } \
>                { eq {0}{${lookup mysql{}{$value}fail}}} \
>               } {X-Spam-Score::X-Spam-Report} }
> #=============

>
> In both cases, the emails were delivered as expected.
>
> Therefore, I think it is a stronger possibility that somehow exim is
> unable to expand the section inside lookup mysql{......}. Given that the
> query is correct (I tried with fixed values from command line in mysql, and
> they both came up with an empty value (in other words, "No rows."). So I
> think it is safe to check if the "fail" is actually working, or how is that
> fail behaving with respect to the or {...}. But then again, not being a
> coder myself I might be completely off track -- I'll leave the debugging to
> you.
>
> If you'd like me to run any other test, please do let me know, and I'll
> see how much I can do.
>
> Thanks again
> Shah
>
>
> On 5 September 2014 13:42, Shamim Shahriar <shamim.shahriar@???>
> wrote:
>
>> Thanks Jeremy, I revisited the conditions, and once I got rid of all the
>> text inside the brackets, this is what the barebone version comes to
>>
>> ${if or
>>        {
>>           { <{ } { } } \
>>           { <{ }{ ${mysql{ '${}' '${}' }{} }} } \
>>           { {0}{${mysql{ '${}' '${}' }{} }}} \
>>        } {X-Spam-Score::X-Spam-Report}
>> }

>>
>> As you can see, there is a closing bracket right before the if-result,
>> which is against the "or{". Also, the three conditions inside OR { } does
>> begin and close where expected.
>>
>> However, in order to make sure that I am not miscalculating, I added one
>> } right where you pointed, and below are the results
>>
>> 07:49:13 24368 failed to expand: select users.sa_tag * 10 from
>> users,domains where local
>> 07:49:13 24368    error message: missing } at end of string
>> 07:49:13 24368 failed to expand: ${lookup mysql{select users.sa_tag * 10
>> from users,domains where local
>> 07:49:13 24368    error message: missing } at end of string
>> 07:49:13 24368 failed to expand: ${if or { { <{$spam_score_int}{1} } {
>> <{$spam_score_int}{${lookup mysql{select users.sa_tag * 10 from
>> users,domains where local
>> 07:49:13 24368    error message: missing } at end of string inside
>> "or{...}" condition
>> 07:49:13 24368 virtual_domains router: defer for
>> postmaster@???
>> 07:49:13 24368   message: virtual_domains router failed to expand "${if
>> or { { <{$spam_score_int}{1} } { <{$spam_score_int}{${lookup mysql{select
>> users.sa_tag * 10 from users,domains where localpart =
>> '${quote_mysql::$local_part}' and domain = '${quote_mysql::$domain}' and
>> users.on_spamassassin = 1 and
>> users.domain_id=domains.domain_id}{$value}fail}} } { eq {0}{${lookup
>> mysql{select users.sa_tag * 10 from users,domains where localpart =
>> '${quote_mysql::$local_part}' and domain = '${quote_mysql::$domain}' and
>> users.on_spamassassin = 0 and
>> users.domain_id=domains.domain_id}{$value}fail}}} }     }
>> {X-Spam-Score::X-Spam-Report} }": missing } at end of string inside
>> "or{...}" condition

>>
>>
>> As it seems, it is still thinking there is one missing curly bracket.
>> Strange!
>>
>> Any other thoughts?
>>
>> Thanks again for your help.
>>
>> Regards
>> Shah
>>
>>
>> On 5 September 2014 12:37, Jeremy Harris <jgh@???> wrote:
>>
>>> On 05/09/14 12:10, Shamim Shahriar wrote:
>>> > Hello Jeremy
>>> >
>>> > Thank you for your reply. I tried that, and below is the modified
>>> > headers_remove section
>>> >
>>> >   headers_remove = ${if or { { <{$spam_score_int}{1} } \
>>> >         { <{$spam_score_int}{${lookup mysql{select users.sa_tag * 10
>>> from
>>> > users,domains \
>>> >         where localpart = '${quote_mysql::$local_part}' \
>>> >         and domain = '${quote_mysql::$domain}' \
>>> >         and users.on_spamassassin = 1 \
>>> >         and users.domain_id=domains.domain_id}{$value}fail}} } \
>>> >         { eq {0}{${lookup mysql{select users.sa_tag * 10 from
>>> users,domains
>>> > \
>>> >         where localpart = '${quote_mysql::$local_part}' \
>>> >         and domain = '${quote_mysql::$domain}' \
>>> >         and users.on_spamassassin = 0 \
>>> >         and users.domain_id=domains.domain_id}{$value}fail}}} \
>>> >         } {X-Spam-Score::X-Spam-Report} }

>>>
>>> If I'm counting brackets right, you have
>>>
>>>  {if
>>>    or {    { cond1 }
>>>            { cond2 }
>>>            { cond3 }

>>>
>>>                               <<<<<<<<<< MISSING

>>>
>>> { if-result }
>>>
>>> }
>>>
>>> --
>>> Cheers,
>>> Jeremy
>>>
>>> --
>>> ## List details at https://lists.exim.org/mailman/listinfo/exim-users
>>> ## Exim details at http://www.exim.org/
>>> ## Please use the Wiki with this list - http://wiki.exim.org/
>>>
>>
>>
>