[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Re: Even more test code

From: Mo DeJong <mdejong_at_cygnus.com>
Date: 2001-04-27 08:59:57 CEST

On Fri, 27 Apr 2001, Sam TH wrote:

> Ok, in additon to the changes mentioned in my last email, this has
> even more new code. =20
>
> - I created a couple basic tests in the basic_tests.py file. They're
> pretty boring.
>
> - I added support for the concept of expected failures in our test
> suite. We now report one four possible results for a test.
> =20
> PASS - the test was expected to pass, and passed
> FAIL - the test was expected to pass, and failed
> XPASS - the test was expected to fail, and passed
> XFAIL - the test was expected to fail, and failed
>
> Note that the naming doesn't quite make sense - unexpected passes are
> flagged, and unexpected failures aren't. But this is the way
> everybody else does it. If you have a better naming scheme, I'm all
> ears.

I would suggest that you avoid using this scheme. This is the
DejaGnu style, and to be honest I think it is really confusing.
Instead, how about the following results:

PASSED
FAILED
SKIPPED
COREDUMP

The PASSED result means the test case result matched
the expected result. A FAILED result means the test
case result did not match the expected result. When
you see a result transition like:

PASSED -> FAILED

It is clear that this is a regression.

Same goes for:

PASSED -> COREDUMP

This is not the same as a FAILED since we would
want to take special note of a code path that
drops core. Along the same lines, we would want
to be informed of a transition like:

FAILED -> COREDUMP

That is clearly not good. The other side of
the coin is a transition like:

FAILED -> PASSED

It is clear that you just fixed a bug.
This approach is a lot easier to grasp IMHO.

Mo DeJong
Red Hat Inc
Received on Sat Oct 21 14:36:29 2006

This is an archived mail posted to the Subversion Dev mailing list.