On Thu, 2010-01-07 at 10:20 +0800, Ed wrote:
> Julian Foad wrote:
> > The report of "success" indicates a bug in the test harness. We had (and
> > still have) similar problems with the Linux test harness.
>
> I'm still having slight difficulties understanding these tests
> in the regards that an XFAIL is a success despite it being
> expected to fail.
See the description of XFAIL in the section "How to write new C tests"
in
<http://svn.apache.org/repos/asf/subversion/trunk/subversion/tests/README>. XFAIL means "We know this test fails, and are happy to live with that for some longish time." At the level of running a whole suite of tests, we want to consider the result as successful if the only failures were expected failures, so that's what the test harness does.
> But for some reasons, Linux tests don't report svn segfaulting,
When you say they "don't report it", what they should do is catch svn
segfaulting and treat that as a test failing, the same as if svn
produced any kind of unexpected results. If the test in which
svnsegfaults is marked "XFAIL", then it should report the result as
"XFAIL" and therefore effectively a pass.
Sometimes a test marks XFAIL fails for a reason other than the expected
reason - e.g. it segfaults whereas the original expected failure was
some particular unwanted commandline output. We don't have a way of
detecting this "fails for the wrong reason" scenario.
> unlike Windows tests which svn.exe crashes. Guess I should
> mark that as an operating system difference? Anyway, thanks
> for the clarification.
- Julian
Received on 2010-01-07 13:29:28 CET