C. Michael Pilato wrote:
> Bert Huijben wrote:
>
>>> -----Original Message-----
>>> From: Branko Cibej [mailto:brane_at_xbc.nu]
>>> Sent: dinsdag 10 maart 2009 16:27
>>> To: Branko Cibej; dev_at_subversion.tigris.org
>>> Subject: Re: Test XFail/Skip policy
>>>
>>> Stefan Sperling wrote:
>>>
>>>> On Tue, Mar 10, 2009 at 04:07:15PM +0100, Branko Cibej wrote:
>>>>
>>>>
>>>>> The thing I'd find useful is adding an optional comment to XFail and
>>>>> Skip; so for this test, you could Xfail(foo, reason="yeah we know its
>>>>> broken, this is issue #bla, foo@ is working on it, don't panic")
>>>>>
>>>>>
>>>> Yeah, that would do!
>>>>
>>>>
>>> Guess what -- that's a bitesize (for me). :) And an opportunity to
>>> contribute some code, not just blab, after a long time. I'm on it.
>>>
>> I think we need a separate marker for might fail, and must fail.
>>
>> Currently XFail is a must-fail and breaks the buildbots if it doesn't fail
>> (XPass error).
>>
>
> I assert that there is no such thing as "might fail". If you know exactly
> what situations cause a test to fail, then test those conditions and claim
> that the test must fail when those conditions are true (XFail). If you
> *don't* know what situations cause a test to fail, that's a bug and needs to
> be flagged as such with an unexpected failure.
>
I agree. You (Bert) have also not yet explained why you think it's good
for tests to fail in a way that our buildbots don't notice -- is it just
about not receiving a bunch of failed-test mails?
We're not reinventing the wheel WRT test results here; we inherited the
PASS/FAIL/XPASS/XFAIL from older systems that have a long history of
using them.
-- Brane
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=1303561
Received on 2009-03-10 16:41:36 CET