Re: [exim-dev] test suite out of order results

On Tue, May 1, 2012 at 6:36 AM, Todd Lyons <tlyons@???> wrote:
> When I run the test suite, some tests give me out of order results in
> the log file.  A good example is test #5.  Ever seen this before?  Can
> anybody suggest changes I can or should make to get these log lines
> appearing in the right order?  This is running on a real machine, not

Jeremy Harris caught up to me in IRC and figured it out. User
"exim-build" is where my git checkout and my test framework is
located. User "exim" is what Local/Makefile expects the config and
spool directories to be owned by (EXIM_USER and EXIM_GROUP). I had
incorrectly also configured CONFIGURE_OWNER and CONFIGURE_GROUP to
"exim". Once I changed it to "exim-build", rebuilt the binary, and
re-ran the test, my system successfully runs test 5 now.

So a FAQ item for running the test suite:
Q: Why does running the test suite fail on test 0005 with the log
lines all present, but just out of order?
A: The config file that the test suite uses is owned by a different
user than what is configured in the exim version the test suite is
testing. If you look at the stderr output log file, you'll likely see
it's complaining about the ownership of the config file. Modify the
src/Local/Makefile, setting CONFIGURE_OWNER and CONFIGURE_GROUP to the
user you are running the test as, rebuild exim, and repeat the test.

** I don't know _why_ that caused the problem, but I do know that
changing the settings _fixed_ the problem. **

Next step is to figure out why test #22 is failing for me. For some
reason my exim is "using cached result" when looking up password data,
but the stderr/0022 expects "cache not available". More testing...
:-)

...Todd
--
Always code as if the guy who ends up maintaining your code will be a
violent psychopath who knows where you live. -- Martin Golding