[exim-dev] Need feedback on a test-suite test

Top Page
Delete this message
Reply to this message
Author: John Jetmore
Date:  
To: exim-dev
Subject: [exim-dev] Need feedback on a test-suite test
There are a couple of Exim tests that involve log lines changing order
from the previously-expected output. Here's the expected output and
then the current output (not showing diffs because this is one
situation where diff's output is really misleading):

TEST 0358
EXPECTED OUTPUT:
+++++++++++++++++++++++++++
R:userx@???:<CALLER@???> -44 13121 SMTP error from remote
mail server after RCPT TO:<userx@???>: host 127.0.0.1 [127.0.0.1]:
451 Temporary error
first failed = time last try = time2 next try = time2 + 1
R:usery@???:<CALLER@???> -44 13121 SMTP error from remote
mail server after RCPT TO:<usery@???>: host 127.0.0.1 [127.0.0.1]:
451 Temporary error
first failed = time last try = time2 next try = time2 + 1
+++++++++++++++++++++++++++
R:userx@???:<CALLER@???> -44 13121 SMTP error from remote
mail server after RCPT TO:<userx@???>: host 127.0.0.1 [127.0.0.1]:
451 Temporary error
first failed = time last try = time2 next try = time2 + 2
R:usery@???:<CALLER@???> -44 13121 SMTP error from remote
mail server after RCPT TO:<usery@???>: host 127.0.0.1 [127.0.0.1]:
451 Temporary error
first failed = time last try = time2 next try = time2 + 2
ACTUAL OUTPUT:
+++++++++++++++++++++++++++
R:usery@???:<CALLER@???> -44 13121 SMTP error from remote
mail server after RCPT TO:<usery@???>: host 127.0.0.1 [127.0.0.1]:
451 Temporary error
first failed = time last try = time2 next try = time2 + 1
R:userx@???:<CALLER@???> -44 13121 SMTP error from remote
mail server after RCPT TO:<userx@???>: host 127.0.0.1 [127.0.0.1]:
451 Temporary error
first failed = time last try = time2 next try = time2 + 1
+++++++++++++++++++++++++++
R:usery@???:<CALLER@???> -44 13121 SMTP error from remote
mail server after RCPT TO:<usery@???>: host 127.0.0.1 [127.0.0.1]:
451 Temporary error
first failed = time last try = time2 next try = time2 + 2
R:userx@???:<CALLER@???> -44 13121 SMTP error from remote
mail server after RCPT TO:<userx@???>: host 127.0.0.1 [127.0.0.1]:
451 Temporary error
first failed = time last try = time2 next try = time2 + 2

TEST 0548
EXPECTED OUTPUT
+++++++++++++++++++++++++++
R:userx@???:<CALLER@???> -44 13121 SMTP error from remote
mail server after RCPT TO:<userx@???>: host ipv4.ipv4.ipv4.ipv4
[ipv4.ipv4.ipv4.ipv4]: 451 Recipient deferred
first failed = time last try = time2 next try = time2 + 1
T:thishost.test.ex:127.0.0.1:1225 0 65 SMTP error from remote mail
server after initial connection: host thishost.test.ex [127.0.0.1]:
451 host deferred
first failed = time last try = time2 next try = time2 + 1
+++++++++++++++++++++++++++
R:userx@???:<CALLER@???> -44 13121 SMTP error from remote
mail server after RCPT TO:<userx@???>: host ipv4.ipv4.ipv4.ipv4
[ipv4.ipv4.ipv4.ipv4]: 451 Recipient deferred
first failed = time last try = time2 next try = time2 + 5 *
T:thishost.test.ex:127.0.0.1:1225 0 65 SMTP error from remote mail
server after initial connection: host thishost.test.ex [127.0.0.1]:
451 host deferred
first failed = time last try = time2 next try = time2 + 1
+++++++++++++++++++++++++++
R:userx@???:<CALLER@???> -44 13121 SMTP error from remote
mail server after RCPT TO:<userx@???>: host ipv4.ipv4.ipv4.ipv4
[ipv4.ipv4.ipv4.ipv4]: 451 Recipient deferred
first failed = time last try = time2 next try = time2 + 5 *
T:thishost.test.ex:127.0.0.1:1225 0 65 SMTP error from remote mail
server after initial connection: host thishost.test.ex [127.0.0.1]:
451 host deferred
first failed = time last try = time2 next try = time2 + 1
ACTUAL OUTPUT:
++++++++++++++++++++++++++
T:thishost.test.ex:127.0.0.1:1225 0 65 SMTP error from remote mail
server after initial connection: host thishost.test.ex [127.0.0.1]:
451 host deferred
first failed = time last try = time2 next try = time2 + 1
R:userx@???:<CALLER@???> -44 13121 SMTP error from remote
mail server after RCPT TO:<userx@???>: host ipv4.ipv4.ipv4.i
pv4 [ipv4.ipv4.ipv4.ipv4]: 451 Recipient deferred
first failed = time last try = time2 next try = time2 + 1
+++++++++++++++++++++++++++
T:thishost.test.ex:127.0.0.1:1225 0 65 SMTP error from remote mail
server after initial connection: host thishost.test.ex [127.0.0.1]:
451 host deferred
first failed = time last try = time2 next try = time2 + 1
R:userx@???:<CALLER@???> -44 13121 SMTP error from remote
mail server after RCPT TO:<userx@???>: host ipv4.ipv4.ipv4.ipv4
[ipv4.ipv4.ipv4.ipv4]: 451 Recipient deferred
first failed = time last try = time2 next try = time2 + 5 *
+++++++++++++++++++++++++++
T:thishost.test.ex:127.0.0.1:1225 0 65 SMTP error from remote mail
server after initial connection: host thishost.test.ex [127.0.0.1]:
451 host deferred
first failed = time last try = time2 next try = time2 + 1
R:userx@???:<CALLER@???> -44 13121 SMTP error from remote
mail server after RCPT TO:<userx@???>: host ipv4.ipv4.ipv4.ipv4
[ipv4.ipv4.ipv4.ipv4]: 451 Recipient deferred
first failed = time last try = time2 next try = time2 + 5 *

Sorry for the wall of text.

My concern is that this change in line ordering is perhaps an
indication of a larger problem and I don't see it. Does anyone know
why this would have changed and whether it's expected, or at least
tolerable?

This probably won't make sense to anyone who hasn't spent a bunch of
time with the suite, but here's the script that's driving it:

# recipient and host defer interactions
# In this test, the primary host (normal IP address) defers the recipient, and
# the secondary host (loopback address) defers the connection.
#
need_ipv4
#
exim -DSERVER=server -bd -oX PORT_D
****
exim -odi userx@???
Test message
****
dump retry
sleep 2
# At this point, the secondary host should have timed out
exim -q
****
dump retry
sleep 1
exim -q
****
dump retry
exim -odi usery@???
Test message 2.
****
sleep 1
exim -q
****
killdaemon
no_msglog_check

Thanks for any help
--John