> -----Original Message-----
> From: Branko Čibej [mailto:brane_at_xbc.nu]
> Sent: dinsdag 10 maart 2009 16:41
> To: C. Michael Pilato
> Cc: Bert Huijben; dev_at_subversion.tigris.org
> Subject: Re: Test XFail/Skip policy
>
> C. Michael Pilato wrote:
> > Bert Huijben wrote:
> >
> >>> -----Original Message-----
> >>> From: Branko Cibej [mailto:brane_at_xbc.nu]
> >>> Sent: dinsdag 10 maart 2009 16:27
> >>> To: Branko Cibej; dev_at_subversion.tigris.org
> >>> Subject: Re: Test XFail/Skip policy
> >>>
> >>> Stefan Sperling wrote:
> >>>
> >>>> On Tue, Mar 10, 2009 at 04:07:15PM +0100, Branko Cibej wrote:
> >>>>
> >>>>
> >>>>> The thing I'd find useful is adding an optional comment to XFail and
> >>>>> Skip; so for this test, you could Xfail(foo, reason="yeah we know its
> >>>>> broken, this is issue #bla, foo@ is working on it, don't panic")
> >>>>>
> >>>>>
> >>>> Yeah, that would do!
> >>>>
> >>>>
> >>> Guess what -- that's a bitesize (for me). :) And an opportunity to
> >>> contribute some code, not just blab, after a long time. I'm on it.
> >>>
> >> I think we need a separate marker for might fail, and must fail.
> >>
> >> Currently XFail is a must-fail and breaks the buildbots if it doesn't fail
> >> (XPass error).
> >>
> >
> > I assert that there is no such thing as "might fail". If you know exactly
> > what situations cause a test to fail, then test those conditions and claim
> > that the test must fail when those conditions are true (XFail). If you
> > *don't* know what situations cause a test to fail, that's a bug and needs to
> > be flagged as such with an unexpected failure.
> >
>
> I agree. You (Bert) have also not yet explained why you think it's good
> for tests to fail in a way that our buildbots don't notice -- is it just
> about not receiving a bunch of failed-test mails?
The big problem is that a failed mail with 5 errors doesn't stand out between hundreds of mails per day with 4 errors.
If there is at least one error http://www.mobsol.be/buildbot/waterfall and http://crest.ics.uci.edu/buildbot/waterfall are red. So the alarm effect is 0 if this status is kept for days. (We started failing Saturday).
If the build fails here at work, it must be fixed ASAP.. not half a week later.
(These patch tests started failing last Saturday; the moment the branch was merged to trunk)
> We're not reinventing the wheel WRT test results here; we inherited the
> PASS/FAIL/XPASS/XFAIL from older systems that have a long history of
> using them.
This all assumes somebody investigates problems until they are a stable PASS or a stable FAIL, which is not the case here.
I think Arfrever can use some help to get these tests in either an Pass or an XFail. Until that point is reached the test code doesn't belong on trunk. At least not in a way that it can fail the buildbots.
Bert
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=1303784
Received on 2009-03-10 17:10:40 CET