Re: [exim-dev] test suite

Top Page

Reply to this message
Author: Jeremy Harris
To: exim-dev
Subject: Re: [exim-dev] test suite
On 2012-04-16 21:44, Todd Lyons wrote:
> On Mon, Apr 16, 2012 at 1:21 PM, Jeremy Harris<jgh@???> wrote:
>> So, I'm running fairly clean passes of the testsuite now,
>> but I have a fairly long list of testcase output changes needed
>> to do so.
> Can you post a list of what does seem to require changes? I have a
> lot of test cases failing due to strange differences. Here's a good
> example:
> Basic/0002 Common string expansions
> ===============f test-stderr-munged with stderr/0002 failed
> Line 177 of "test-stderr-munged" does not match line 177 of "stderr/0002".
> ----------
>    5TESTSUITE/aux-fixed/0002.lsearch
> ----------
>    :TESTSUITE/aux-fixed/0002.lsearch


> Here's another one, the munged output file seems to be sorted incorrectly maybe?
> Basic/0005 -bs to simple local delivery
> ===============f test-mainlog-munged with log/0005 failed

That one's not on my list. What system are you testing on?

> I've got lots more that are like that.
> If it's too painful to read this with line wrapping, in the future
> I'll use the dreaded rich formatting to keep it looking better.

Personally I prefer plain ascii.

Here's my current list (but I've not filtered out the ones due to 1184 yet,
so use a healthy ping of salt):

#       modified:   log/0027
#       modified:   log/1003
#       modified:   log/3411
#       modified:   rejectlog/0027
#       modified:   rejectlog/3411
#       modified:   stderr/0002
#       modified:   stderr/0038
#       modified:   stderr/0085
#       modified:   stderr/0123
#       modified:   stderr/0387
#       modified:   stderr/0393
#       modified:   stderr/0403
#       modified:   stderr/0414
#       modified:   stderr/0437
#       modified:   stderr/0464
#       modified:   stderr/0471
#       modified:   stderr/0484
#       modified:   stderr/3400
#       modified:   stderr/3408
#       modified:   stderr/3410
#       modified:   stderr/3500
#       modified:   stdout/0240
#       modified:   stdout/0345
#       modified:   stdout/0373
#       modified:   stdout/1006
#       modified:   stdout/1009
#       modified:   stdout/2102
#       modified:   stdout/2103
#       modified:   stdout/2105
#       modified:   stdout/2106
#       modified:   stdout/2114
#       modified:   stdout/2118
#       modified:   stdout/2119
#       modified:   stdout/2122
#       modified:   stdout/2124
#       modified:   stdout/2128
#       modified:   stdout/2150
#       modified:   stdout/3460
#       modified:   stdout/3463
#       modified:   stdout/3464

I'm running local (if virtualised) filesystems. Two users:


eximtest is a member of both groups.
The exim build has EXIM_USER=ref:exim

I've noticed some flakiness in some of the tests;
eventually it'll need tracking down and making solid.
There's something odd in time-related stuff somewhere.

I'm considering a "retry without adjusting results-files"
addition to the runtest script.