Brane, thanks much for the XFAIL stuff. I've just one issue with it:
when "make check" is printing out the summary results, it treats
expected failures as "FAILURE"s. This is needlessly alarming -- the
whole *point* of the all-caps word "FAILURE" is to stand out and let
the programmer know that something's wrong, so don't commit now.
Tests that generate expected failures should print "success". They
behaved as expected, so they succeded.
As for printing a list of the expected failures *after* the summary
(see example below), I'm -0.5 on that. After all, we don't depend on
the test suite to tell us what needs fixing, we depend on the issue
tracker. Seeing a full list of XFAILs at the end of the test run is
just more noise for the programmer to sort through, because the
programmer is interested in *at most* 1 of those failures -- all the
rest are just in the way. (And more often, 0 of the expected failures
will be interesting, because the programmer is working on something
else entirely.)
In summary:
* The test suite is for telling you whether you can fixed a bug or
not, and/or whether you broke something else in the process.
* The issue tracker is for remembering what needs to get fixed, and
for scheduling when we plan to fix it.
:-)
Here's the new "make check" output that just caused me to jump:
Running all tests in hashdump-test...success
Running all tests in path-test...success
[...]
Running all tests in basic_tests.py...success
Running all tests in commit_tests.py...FAILURE
Running all tests in update_tests.py...success
Running all tests in switch_tests.py...success
Running all tests in prop_tests.py...success
Running all tests in schedule_tests.py...FAILURE
Running all tests in log_tests.py...success
[...]
Running all tests in stat_tests.py...success
Running all tests in trans_tests.py...FAILURE
Running all tests in svnadmin_tests.py...success
Summary results from /home/kfogel/src/subversion/tests.log
number of passed tests: 240
number of failed tests: 0
number of expected failures: 9
number of unexpected passes: 0
Expected failures:
XFAIL: commit_tests.py 13: hook testing.
XFAIL: schedule_tests.py 11: commit: add some files
XFAIL: schedule_tests.py 12: commit: add some directories
XFAIL: schedule_tests.py 13: commit: add some nested files and directories
XFAIL: schedule_tests.py 14: commit: delete some files
XFAIL: schedule_tests.py 15: commit: delete some directories
XFAIL: trans_tests.py 2: enable translation, check status, commit
XFAIL: trans_tests.py 3: checkout files that have translation enabled
XFAIL: trans_tests.py 4: disable translation, check status, commit
make: *** [check] Error 1
I'm not objecting to the "Summary results from..." section. It's the
final list of "Expected failures" that bothers me, because it's noise,
and it will only get longer and longer, and drive useful information
off the screen.
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Thu Aug 22 06:47:00 2002