Re: expected failures shouldn't raise alarms

Karl Fogel wrote:

>Branko Čibej <brane@xbc.nu> writes:
>
>
>>Let me try to recapitulate what you want:
>>
>> * Lose the summary section
>> * List only unexpected results (FAIL and XPASS)
>>
>>Right?
>>
>>
>
>Right. (I probably should have included a concise summary in my
>mail.)
>
>
>
>>One more detail: Does an XPASS mean the tests should fail (i.e., exit
>>with non-zero)?
>>
>>
>
>Hmmm. I don't really understand the purpose of XPASS, perhaps you can
>help?
>
>If the idea is this:
>
> A bug is present in Subversion, and we've written an XPASS test
> such that as long as the bug is *present*, the test returns
> success, but as soon as the bug is fixed, the test will "fail".
>
>...then I'd say we should simply reverse the sense of the test's
>return value and make it an XFAIL instead. But maybe XPASS is about
>something else?
>

You don't mark tests as XPASS. Here's what happens:

    You write a test for a new bug, and mark it XFAIL. Later on, you fix
    the bug, and the test passes. BUT you forget to un-XFAIL the test.
    So, the test is expected to fail, but unexpectedly passes -- hence
    XPASS instead of PASS.

So, XPASS is a sort of reminder. It's useful because the bug may have
been fixed inadvertently, or maybe the test infrastructure was changed
such that the XFAIL test suddenly passes even though the bug is still
present.

XPASS can only happen to tests that have been marked as XFAIL.

-- 
Brane Čibej   <brane_at_xbc.nu>   http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org