On Mon, 15 Jan 2007, Ian Eiloart wrote:
> I'm getting the hang of this now. I've created a MacPorts installer, which
> makes me a copy of the test suite minus all the tests. I found a few
> problems doing this:
I'm not surprised! The test suite wasn't designed to be test-free. Hmm.
It must be Somebody's Law that any software one releases always gets
used in a completely unexpected manner by someone, thereby causing
"issues". :-)
> 1. The build would have been easier if the Makefile supported 'make
> install'.
There is no concept of "installing" the test suite - it is supposed to
be run where you unpack it. Partly this is because I expect sysadmins
not to want to make it publicly available, but partly also because there
is a different release with each Exim release, so there is also no
concept of "the current test suite". Rather you have "the test suite
that corresponds to Exim release x.y".
> 2. It would be nice to be able to specify a group to use. Normally I run
> exim under group 'mail', so I had to create the group 'exim'.
As it says in the README
(5) Exim must be built with its user and group specified at build time,
and with certain minimum facilities, namely:
Routers: accept, dnslookup, manualroute, redirect
Transports: appendfile, autoreply, pipe, smtp
Lookups: lsearch
Most Exim binaries will have these included.
Changing Exim's uid/gid at runtime has always been a kludge and doesn't
work well. Or have I misunderstood this issue?
> 3. Having removed all the tests, I found that runtests fails if there is no
> test '0000'. I had been hoping to avoid creating name conflicts.
Well, as I said, I never envisaged what you are doing...
> 4. I'm using Exim with spamc and spamd using a unix socket. runtests
> couldn't find spamd.
There is this comment in the code:
# This test for an active SpamAssassin is courtesy of John Jetmore.
# The tests are hard coded to localhost:783, so no point in making
# this test flexible like the clamav test until the test scripts are
# changed.
so that is "as intended". :-(
> 5. To avoid name conflicts, it would be nice to have a reserved range of
> test numbers, perhaps starting at 10,000. I've decided to use 6xxx for my
> MX config and 7xxx for my MSA config.
All tests above 9000 are special-case, not normally run, and those above
9900 are experimental tests. The problem with going above 9999 is that
the code is written for 4-digit test numbers only.
> 6. It would be nice to be able to configure a default binary to use. I
> wasn't able to figure out where runtests is looking for the binary - I
> assumed it was looking for ../exim/bin/exim
>From the README:
----------------------------------------------------------------------
If you do not supply any arguments to ./runtest, it searches for an Exim
source tree at the same level as the test suite directory. It then looks
for an Exim binary in a "build" directory of that source tree. If there
are several Exim source trees, it chooses the latest version of Exim.
Consider the following example:
$ ls -F /source/exim
exim-4.60/ exim-4.62/ exim-testsuite-x.xx/
A simple ./runtest from within the test suite will use a 4.62 binary if
it finds one, otherwise a 4.60 binary. If a binary cannot be found, the
script prompts for one. Alternatively, you can supply the binary on the
command line:
./runtest /usr/exim/bin/exim
A matching test suite is released with each Exim release; if you use a
test suite that does not match the binary, some tests may fail.
----------------------------------------------------------------------
Remember, this was designed for testing Exim itself, so looking in a
"bin" directory is less likely, because that's an installed binary and
normally you test before you install. :-)
> 7. I'll usually want to run my tests all on the same config file, so it
> would be easier to specify a default config file. Otherwise, I can just
> create tons of links, I suppose.
You can, of course, do what you want by just having a single test script
that contains lots of different calls to Exim. The design of the suite
is that each top-level test has a different config file. All the tests
of that config are then located within the single script.
--
Philip Hazel University of Cambridge Computing Service
Get the Exim 4 book: http://www.uit.co.uk/exim-book
