[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

expected failures shouldn't raise alarms

From: Karl Fogel <kfogel_at_newton.ch.collab.net>
Date: 2002-08-22 06:28:38 CEST

Brane, thanks much for the XFAIL stuff. I've just one issue with it:
when "make check" is printing out the summary results, it treats
expected failures as "FAILURE"s. This is needlessly alarming -- the
whole *point* of the all-caps word "FAILURE" is to stand out and let
the programmer know that something's wrong, so don't commit now.

Tests that generate expected failures should print "success". They
behaved as expected, so they succeded.

As for printing a list of the expected failures *after* the summary
(see example below), I'm -0.5 on that. After all, we don't depend on
the test suite to tell us what needs fixing, we depend on the issue
tracker. Seeing a full list of XFAILs at the end of the test run is
just more noise for the programmer to sort through, because the
programmer is interested in *at most* 1 of those failures -- all the
rest are just in the way. (And more often, 0 of the expected failures
will be interesting, because the programmer is working on something
else entirely.)

In summary:

   * The test suite is for telling you whether you can fixed a bug or
     not, and/or whether you broke something else in the process.

   * The issue tracker is for remembering what needs to get fixed, and
     for scheduling when we plan to fix it.


Here's the new "make check" output that just caused me to jump:

   Running all tests in hashdump-test...success
   Running all tests in path-test...success
   Running all tests in basic_tests.py...success
   Running all tests in commit_tests.py...FAILURE
   Running all tests in update_tests.py...success
   Running all tests in switch_tests.py...success
   Running all tests in prop_tests.py...success
   Running all tests in schedule_tests.py...FAILURE
   Running all tests in log_tests.py...success
   Running all tests in stat_tests.py...success
   Running all tests in trans_tests.py...FAILURE
   Running all tests in svnadmin_tests.py...success
   Summary results from /home/kfogel/src/subversion/tests.log
   number of passed tests: 240
   number of failed tests: 0
   number of expected failures: 9
   number of unexpected passes: 0
   Expected failures:
   XFAIL: commit_tests.py 13: hook testing.
   XFAIL: schedule_tests.py 11: commit: add some files
   XFAIL: schedule_tests.py 12: commit: add some directories
   XFAIL: schedule_tests.py 13: commit: add some nested files and directories
   XFAIL: schedule_tests.py 14: commit: delete some files
   XFAIL: schedule_tests.py 15: commit: delete some directories
   XFAIL: trans_tests.py 2: enable translation, check status, commit
   XFAIL: trans_tests.py 3: checkout files that have translation enabled
   XFAIL: trans_tests.py 4: disable translation, check status, commit
   make: *** [check] Error 1

I'm not objecting to the "Summary results from..." section. It's the
final list of "Expected failures" that bothers me, because it's noise,
and it will only get longer and longer, and drive useful information
off the screen.


To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Thu Aug 22 06:47:00 2002

This is an archived mail posted to the Subversion Dev mailing list.

This site is subject to the Apache Privacy Policy and the Apache Public Forum Archive Policy.