[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Re: expected failures shouldn't raise alarms

From: Karl Fogel <kfogel_at_newton.ch.collab.net>
Date: 2002-08-22 19:10:43 CEST

Branko Čibej <brane@xbc.nu> writes:
> You don't mark tests as XPASS. Here's what happens:
> You write a test for a new bug, and mark it XFAIL. Later on, you fix
> the bug, and the test passes. BUT you forget to un-XFAIL the test.
> So, the test is expected to fail, but unexpectedly passes -- hence
> XPASS instead of PASS.
> So, XPASS is a sort of reminder. It's useful because the bug may have
> been fixed inadvertently, or maybe the test infrastructure was changed
> such that the XFAIL test suddenly passes even though the bug is still
> present.
> XPASS can only happen to tests that have been marked as XFAIL.

Oooooh. I find that a bit confusing, because I think of the "X"
prefix as standing for "eXpected", where as in XPASS it stands for

But whatever. Let's make XPASS exit with non-zero, so it counts as a
failure as far as the final result of the test run goes, for the
reason I gave earlier:

> Btw, yes, I think an unexpected pass should be treated as a kind
> of breakage. If we used to have a bug, and now we don't, then we
> want to notice if it reappears, because that's a regression.
> Therefore, we need to be alerted to the unexpected pass, so we
> can go change the test suite to expect the pass in the future, in
> turn so we'll detect any regression.


To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Thu Aug 22 19:29:54 2002

This is an archived mail posted to the Subversion Dev mailing list.

This site is subject to the Apache Privacy Policy and the Apache Public Forum Archive Policy.