Karl Fogel wrote:
>Brane, thanks much for the XFAIL stuff. I've just one issue with it:
>when "make check" is printing out the summary results, it treats
>expected failures as "FAILURE"s. This is needlessly alarming -- the
>whole *point* of the all-caps word "FAILURE" is to stand out and let
>the programmer know that something's wrong, so don't commit now.
>
>Tests that generate expected failures should print "success". They
>behaved as expected, so they succeded.
>
Perhaps. This isn't just about issue tracking, it's also about enhancing
the test suite. There were 9 tests in thre that were disabled, and most
of those were not even implemented -- which means somebody thought
they'd be a good idea, then forgot about them. There's nothing in the
issue tracker about those tests. :-)
>As for printing a list of the expected failures *after* the summary
>(see example below), I'm -0.5 on that. After all, we don't depend on
>the test suite to tell us what needs fixing, we depend on the issue
>tracker. Seeing a full list of XFAILs at the end of the test run is
>just more noise for the programmer to sort through, because the
>programmer is interested in *at most* 1 of those failures -- all the
>rest are just in the way. (And more often, 0 of the expected failures
>will be interesting, because the programmer is working on something
>else entirely.)
>
I don't print just the XFAILs. I print the FAILs and XPASSes, too. I
think all of those are important. Perhaps it would make sense to print
the XFAIl, XPASS and FAIL lists first, and the summary afterwards? I
don't regard the "Running all tests in ...." stuff as important
information, only the summary and what comes after it.
>In summary:
>
> * The test suite is for telling you whether you can fixed a bug or
> not, and/or whether you broke something else in the process.
>
and whether the test suite is still broken. :-)
> * The issue tracker is for remembering what needs to get fixed, and
> for scheduling when we plan to fix it.
>
But only if you create a new issue for each XFAILed test. There isn't
one now.
>Here's the new "make check" output that just caused me to jump:
>
> Running all tests in hashdump-test...success
> Running all tests in path-test...success
> [...]
> Running all tests in basic_tests.py...success
> Running all tests in commit_tests.py...FAILURE
> Running all tests in update_tests.py...success
> Running all tests in switch_tests.py...success
> Running all tests in prop_tests.py...success
> Running all tests in schedule_tests.py...FAILURE
> Running all tests in log_tests.py...success
> [...]
> Running all tests in stat_tests.py...success
> Running all tests in trans_tests.py...FAILURE
> Running all tests in svnadmin_tests.py...success
>
> Summary results from /home/kfogel/src/subversion/tests.log
> number of passed tests: 240
> number of failed tests: 0
> number of expected failures: 9
> number of unexpected passes: 0
>
> Expected failures:
> XFAIL: commit_tests.py 13: hook testing.
> XFAIL: schedule_tests.py 11: commit: add some files
> XFAIL: schedule_tests.py 12: commit: add some directories
> XFAIL: schedule_tests.py 13: commit: add some nested files and directories
> XFAIL: schedule_tests.py 14: commit: delete some files
> XFAIL: schedule_tests.py 15: commit: delete some directories
> XFAIL: trans_tests.py 2: enable translation, check status, commit
> XFAIL: trans_tests.py 3: checkout files that have translation enabled
> XFAIL: trans_tests.py 4: disable translation, check status, commit
> make: *** [check] Error 1
>
I think only the last line is wrong: If there are only expected
failures, the test driver shouldn't be exiting with a non-zero code.
I'll fix that.
>I'm not objecting to the "Summary results from..." section. It's the
>final list of "Expected failures" that bothers me, because it's noise,
>and it will only get longer and longer, and drive useful information
>off the screen.
>
>
I say, if that list will get longer and longer, then I should print it
in blinking mode. It's there as an incentive for fixing the test suite.
--
Brane Čibej <brane_at_xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Thu Aug 22 11:51:48 2002