Branko Čibej <brane@xbc.nu> writes:
> You don't mark tests as XPASS. Here's what happens:
>
> You write a test for a new bug, and mark it XFAIL. Later on, you fix
> the bug, and the test passes. BUT you forget to un-XFAIL the test.
> So, the test is expected to fail, but unexpectedly passes -- hence
> XPASS instead of PASS.
>
> So, XPASS is a sort of reminder. It's useful because the bug may have
> been fixed inadvertently, or maybe the test infrastructure was changed
> such that the XFAIL test suddenly passes even though the bug is still
> present.
>
> XPASS can only happen to tests that have been marked as XFAIL.
Oooooh. I find that a bit confusing, because I think of the "X"
prefix as standing for "eXpected", where as in XPASS it stands for
"uneXpected".
But whatever. Let's make XPASS exit with non-zero, so it counts as a
failure as far as the final result of the test run goes, for the
reason I gave earlier:
> Btw, yes, I think an unexpected pass should be treated as a kind
> of breakage. If we used to have a bug, and now we don't, then we
> want to notice if it reappears, because that's a regression.
> Therefore, we need to be alerted to the unexpected pass, so we
> can go change the test suite to expect the pass in the future, in
> turn so we'll detect any regression.
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Thu Aug 22 19:29:54 2002