[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Re: expected failures shouldn't raise alarms

From: Branko Čibej <brane_at_xbc.nu>
Date: 2002-08-22 19:34:10 CEST

Karl Fogel wrote:

>Branko Čibej <brane@xbc.nu> writes:
>>You don't mark tests as XPASS. Here's what happens:
>> You write a test for a new bug, and mark it XFAIL. Later on, you fix
>> the bug, and the test passes. BUT you forget to un-XFAIL the test.
>> So, the test is expected to fail, but unexpectedly passes -- hence
>> XPASS instead of PASS.
>>So, XPASS is a sort of reminder. It's useful because the bug may have
>>been fixed inadvertently, or maybe the test infrastructure was changed
>>such that the XFAIL test suddenly passes even though the bug is still
>>XPASS can only happen to tests that have been marked as XFAIL.
>Oooooh. I find that a bit confusing, because I think of the "X"
>prefix as standing for "eXpected", where as in XPASS it stands for
Yup, it's confusing, but it's more or less standard (IIRC it's even a
POSIX stantard for test suites, have too look it up -- the dejagnu
documentation says something about that).

>But whatever. Let's make XPASS exit with non-zero, so it counts as a
>failure as far as the final result of the test run goes, for the
>reason I gave earlier:
> > Btw, yes, I think an unexpected pass should be treated as a kind
> > of breakage. If we used to have a bug, and now we don't, then we
> > want to notice if it reappears, because that's a regression.
> > Therefore, we need to be alerted to the unexpected pass, so we
> > can go change the test suite to expect the pass in the future, in
> > turn so we'll detect any regression.
Right. I'm on it.

Brane Čibej   <brane_at_xbc.nu>   http://www.xbc.nu/brane/
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Thu Aug 22 19:34:51 2002

This is an archived mail posted to the Subversion Dev mailing list.

This site is subject to the Apache Privacy Policy and the Apache Public Forum Archive Policy.