Karl Fogel wrote:
>Branko Čibej <brane@xbc.nu> writes:
>
>
>>You don't mark tests as XPASS. Here's what happens:
>>
>> You write a test for a new bug, and mark it XFAIL. Later on, you fix
>> the bug, and the test passes. BUT you forget to un-XFAIL the test.
>> So, the test is expected to fail, but unexpectedly passes -- hence
>> XPASS instead of PASS.
>>
>>So, XPASS is a sort of reminder. It's useful because the bug may have
>>been fixed inadvertently, or maybe the test infrastructure was changed
>>such that the XFAIL test suddenly passes even though the bug is still
>>present.
>>
>>XPASS can only happen to tests that have been marked as XFAIL.
>>
>>
>
>Oooooh. I find that a bit confusing, because I think of the "X"
>prefix as standing for "eXpected", where as in XPASS it stands for
>"uneXpected".
>
Yup, it's confusing, but it's more or less standard (IIRC it's even a
POSIX stantard for test suites, have too look it up -- the dejagnu
documentation says something about that).
>But whatever. Let's make XPASS exit with non-zero, so it counts as a
>failure as far as the final result of the test run goes, for the
>reason I gave earlier:
>
> > Btw, yes, I think an unexpected pass should be treated as a kind
> > of breakage. If we used to have a bug, and now we don't, then we
> > want to notice if it reappears, because that's a regression.
> > Therefore, we need to be alerted to the unexpected pass, so we
> > can go change the test suite to expect the pass in the future, in
> > turn so we'll detect any regression.
>
>
Right. I'm on it.
--
Brane Čibej <brane_at_xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Thu Aug 22 19:34:51 2002