[exim-dev] Test suite status for 4.77

Góra strony
Delete this message
Reply to this message
Autor: Phil Pennock
Data:  
Dla: exim-dev
Temat: [exim-dev] Test suite status for 4.77
Being of a heretical nature, I once more plunged into the test suite to
see what needed changing to keep the suite mostly working this time.

Being stupid, I did _not_ drink alcohol first.

0152 and 0437 are my main concerns.
On balance, I'm not holding up the 4.77 release for 0152, just 0437
until I understand what is happening.

 * Some improved runtest resiliency to setup problems, to make it
   clearer what might have been missed in setup (not a change)
 * the revamped versioning was not reflected in the tests; updated
   patchexim to handle it
 * the new ratelimit features need test 0038 updated to handle the new
   output and try testing those features; I've mailed Tony, rather than
   tackle that myself tonight, so left 0038 failing.
 * Changed runtest to ignore messages about loading modules
 * Apparently when I added extra tests for boolean inversion in 4.76
   PP/08 I didn't actually update the expected-output file.  Oops.
   Fixed.  (On bright side: I did update the test when I fixed
   something)
 * My usual "6TESTSUITE/aux-fixed/<nnnn>.data" vs ":TESTSUITE..." glitch
   which I need to investigate again, to see what I missed before; for
   now, I've again just continued past those
   [...]
   Ah, ':' the 11th lookup type, with the cache keying using offsets
   from ASCII '0', and ord(':') - ord('0') == 10; I probably just don't
   have the precise set of lookups loaded into the binary that the tests
   prefer.
 * 0152: Extra Sender: header inserted??
   Test comment is: # control of Sender: (local_from_{suf,pre}fix)
   So this is all about Sender: headers and something has changed in the
   three tests from 4th last to 2nd last, to cause a Sender: header to
   be inserted.
   I can't see anything in ChangeLog which would explain this.
   Still broken.
 * 0288: extra messages:
   Berkeley DB error: no absolute path for the current directory: No such file or directory
   Needs investigating, but looks sufficiently like a test suite
   artifact that I'm not looking now.
 * 0358: order inversion of userx/usery results; ignored.
 * 0369: looks like we're fixing up /bin/sh -> SHELL but log/0369
   doesn't know about it?  Consensus change expected output?
   Note though that nothing has changed in fixup here.
 * Basic/0437 queue run: close lookups before delivery
----------------------------8< cut here >8------------------------------
** Command 1 ("exim", starting at line 2)
** Return code 1 (expected 0)
show stdErr, show stdOut, Continue (without file comparison), or Quit? [Q] e
2011-10-10 02:35:23 failed to expand spool_directory "[...]/${lookup{spool}lsearch{[...]/aux-fixed/0437.ls}}": unknown lookup type "lsearch"
----------------------------8< cut here >8------------------------------
   Hrm.  I'd be more worried if I weren't using lsearch in my own setup,
   with every mail sent from pdp@??? using lsearch to get the data
   to authenticate to tahini.
   Also, there are lsearch tests in 0002, 0064, 0219, ...
   If I invoke the binary manually, the failure recurs, but if I pass -d
   then the test passes and I see lsearch working.
 * 0548: order inversion, same issue as 0358
 * 1003: timeout talking to ::1 ... note that this might be further than
   I've gotten in the test suite before, so not sure that it's new.
   I retried and it worked, so looks like it's just flaky.
 * 1006: trailing whitespace difference!  stdout/1006 expects an extra
   25 trailing whitespace characters (26 total, there is actually one
   normally); no idea why that was expected, think that I'm now
   thoroughly into weed-whacking territory
 * 1009: ditto
 * 2102: OpenSSL output variance; lots of diffs reported but it boils
   down to this actual vs observed:
----------------------------8< cut here >8------------------------------
output:
     29 SSL info: SSLv3 read server session ticket A
[...]
     82 SSL info: SSLv3 read server session ticket A
     83 SSL info: SSLv3 read server session ticket A
[...]
    120 SSL info: SSLv3 read server session ticket A
expected:
     81 SSL info: SSLv3 read finished A
     82 SSL info: SSLv3 read finished A
----------------------------8< cut here >8------------------------------
   My recollection (backed by checking!  woot!) is that these status
   notes are provided by the OpenSSL library itself, so I am fully
   prepared to believe that this is a 1.0.0<x> vs 0.9.<whatever>
   difference.
   Ignoring for now, but would appreciate a report from someone else on
   this, so we know whether or not to update the expected output.
 * 2103: 2 extra "SSLv3 read server session ticket A"
 * 2105: 1 extra "SSLv3 read server session ticket A"
 * 2106: 1 extra "SSLv3 read server session ticket A"
 * 2114: many such variances
 * 2118: 1 such variance
 * 2119: 1 such variance
 * 2122: 1 such variance
 * 2124: 2 straight "server session ticket" vs "finished" variances
 * 2128: 2 extra "server session ticket" messages
 * 2150: 1 extra
 * 3460: just when I thought I was free, 1 extra
 * 3463: 2 extra
 * 3464: 2 extra


* 3650: bombed out test suite, we have a script with no config
available; anyone who uses Dovecot willing to volunteer to write the
confs/3650 file?

   Resumed with:
    ./runtest $PWD/../src/build-FreeBSD-amd64/exim 3651 8999


 * 5007 (maildir): Date: header positioning change
----------------------------8< cut here >8------------------------------
4.73:
PP/01 Date: & Message-Id: revert to normally being appended to a message,
      only prepend for the Resent-* case.  Fixes regression introduced in
      Exim 4.70 by NM/22 for Bugzilla 607.
----------------------------8< cut here >8------------------------------
   I remember fixing one other case of this, years ago; I'm clearly
   further on than ever in the test suit to be seeing this only now.


expected output updated

Aaand: test suit completed!
(for the modules tested in my config).