On Tue, Mar 10, 2009 at 02:41:15PM +0100, Branko Cibej wrote:
> Bert Huijben wrote:
> >> -----Original Message-----
> >> From: Arfrever Frehtes Taifersar Arahesis [mailto:arfrever.fta_at_gmail.com]
> >> Temporary XFail would be better than Skip.
> >>
> >
> > I suggested marking them Xfail, but gstein warned against it, as on some linux environments it works and on some it doesn't.
> >
> > Marking it as XFail will give XPass errors on buildbots where it passed before.
> >
>
> Isn't the whole point of the test suite to identify bugs? So, if you
> unconditionally Skip a test that fails on some platforms, it'll never
> get fixed because it'll never get run, and no-one ever reviews skipped
> tetss. Might as well remove the test then, what's the point?
>
> Or maybe fix the bug, who knows.
I concur. What's wrong with a test that occasionally fails if it is
known to occasionally fail because a bug has not been found yet?
As long as people know that this particular test is not failing
because of their own changes, that's fine.
The point of the test suite shouldn't be "all tests must pass all
the time no matter what".
If a test is known to fail because of a particular commit, and we
don't want to annoy people with a failing test, we could back out
the commit and the test, and commit a working combination once the
problem has been found. I've done this before.
Stefan
Received on 2009-03-10 14:56:34 CET