Re: Test XFail/Skip policy

Bert Huijben wrote:
>> -----Original Message-----
>> From: Branko Cibej [mailto:brane_at_xbc.nu]
>> Sent: dinsdag 10 maart 2009 16:27
>> To: Branko Cibej; dev_at_subversion.tigris.org
>> Subject: Re: Test XFail/Skip policy
>>
>> Stefan Sperling wrote:
>>> On Tue, Mar 10, 2009 at 04:07:15PM +0100, Branko Cibej wrote:
>>>
>>>> The thing I'd find useful is adding an optional comment to XFail and
>>>> Skip; so for this test, you could Xfail(foo, reason="yeah we know its
>>>> broken, this is issue #bla, foo@ is working on it, don't panic")
>>>>
>>> Yeah, that would do!
>>>
>> Guess what -- that's a bitesize (for me). :) And an opportunity to
>> contribute some code, not just blab, after a long time. I'm on it.
>
> I think we need a separate marker for might fail, and must fail.
>
> Currently XFail is a must-fail and breaks the buildbots if it doesn't fail
> (XPass error).

I assert that there is no such thing as "might fail". If you know exactly
what situations cause a test to fail, then test those conditions and claim
that the test must fail when those conditions are true (XFail). If you
*don't* know what situations cause a test to fail, that's a bug and needs to
be flagged as such with an unexpected failure.

-- 
C. Michael Pilato <cmpilato_at_collab.net>
CollabNet   <>   www.collab.net   <>   Distributed Development On Demand
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=1303560