On Fri, 27 Apr 2001, Sam TH wrote:
> Ok, in additon to the changes mentioned in my last email, this has
> even more new code. =20
>
> - I created a couple basic tests in the basic_tests.py file. They're
> pretty boring.
>
> - I added support for the concept of expected failures in our test
> suite. We now report one four possible results for a test.
> =20
> PASS - the test was expected to pass, and passed
> FAIL - the test was expected to pass, and failed
> XPASS - the test was expected to fail, and passed
> XFAIL - the test was expected to fail, and failed
>
> Note that the naming doesn't quite make sense - unexpected passes are
> flagged, and unexpected failures aren't. But this is the way
> everybody else does it. If you have a better naming scheme, I'm all
> ears.
I would suggest that you avoid using this scheme. This is the
DejaGnu style, and to be honest I think it is really confusing.
Instead, how about the following results:
PASSED
FAILED
SKIPPED
COREDUMP
The PASSED result means the test case result matched
the expected result. A FAILED result means the test
case result did not match the expected result. When
you see a result transition like:
PASSED -> FAILED
It is clear that this is a regression.
Same goes for:
PASSED -> COREDUMP
This is not the same as a FAILED since we would
want to take special note of a code path that
drops core. Along the same lines, we would want
to be informed of a transition like:
FAILED -> COREDUMP
That is clearly not good. The other side of
the coin is a transition like:
FAILED -> PASSED
It is clear that you just fixed a bug.
This approach is a lot easier to grasp IMHO.
Mo DeJong
Red Hat Inc
Received on Sat Oct 21 14:36:29 2006