Branko Čibej <brane@xbc.nu> writes:
> I say, if that list will get longer and longer, then I should print it
> in blinking mode. It's there as an incentive for fixing the test suite.
No :-). Seriously, -1, although I understand where you're coming from.
We discussed the "make check" output thoroughly when we were
originally designing it, and quite consciously decided that the point
of the output was to tell the programmer if he'd changed anything that
he didn't intend to change. Hence the lower case "success" vs
upper-case "FAILURE", and the list of any *unexpected* failures
following the foo.py summary list.
This was helpful, because it meant that you could glance at the
results and see immediately if you have to dig any deeper. Everything
looks okay? Good, then commit. Something looks wrong? Fine, then
dig deeper, glance over the failure summary, maybe look in tests.log,
go run the failing test(s) by hand, whatever.
The important feature is that the common case, success, could be
distinguished instantaneously from the messy failure case.
Now you've gone and changed all that, by printing an arbitrary number
of expected/unexpected foo's at the end:
Expected failures:
XFAIL: commit_tests.py 13: hook testing.
XFAIL: schedule_tests.py 11: commit: add some files
XFAIL: schedule_tests.py 12: commit: add some directories
XFAIL: schedule_tests.py 13: commit: add some nested files and directories
XFAIL: schedule_tests.py 14: commit: delete some files
XFAIL: schedule_tests.py 15: commit: delete some directories
XFAIL: trans_tests.py 2: enable translation, check status, commit
XFAIL: trans_tests.py 3: checkout files that have translation enabled
XFAIL: trans_tests.py 4: disable translation, check status, commit
What is the justification for this section? That it will somehow
remind developers what needs fixing? Again, the issue tracker is
better for this (and note that just because you don't see the xfail
test listed in an issue summary doesn't mean it's not mentioned in the
issue's description!). Seeing a list of XFAILs is not likely to
suddenly prompt anyone to suddenly decide to work on schedule_test 14,
out of the blue. There's so much information missing here, about
severity and urgency and expected difficulty of fixing... All of which
is probably included in the issue tracker, or should be (i.e., if it's
not, then we should put it there).
This new section(s) is just going to get longer and longer, and make
it impossible to parse the "make check" output in a glance. Heck,
it's already made it impossible to parse in a glance, but at least
we're only up to two glances so far.
Yuck. Ick. Blecch. Please, change it back to the way it was.
Also, Mike Pilato and I were discussing this section
Summary results from /home/kfogel/src/subversion/tests.log
number of passed tests: 240
number of failed tests: 0
number of expected failures: 9
number of unexpected passes: 0
and we frankly can't see what useful information it gives. Why does
it matter how many total tests were in each category? The only
categories that matter up there are the "failed tests" and the
"unexpected passes", and we'd want to signal such events in a much
more attention-getting way if they happen, not as part of routine
summary results.
(Btw, yes, I think an unexpected pass should be treated as a kind of
breakage. If we used to have a bug, and now we don't, then we want to
notice if it reappears, because that's a regression. Therefore, we
need to be alerted to the unexpected pass, so we can go change the
test suite to expect the pass in the future, in turn so we'll detect
any regression.)
My understanding of the purpose of the XFAIL stuff is that it gives
volunteers a really convenient way to take on a bug. The process is
1) Someone finds a bug.
2) Someone writes an XFAIL regression test for it.
3) We file an issue, mentioning the test by name.
4) Someone fixes the bug, looking to make the test "unexpectedly pass"
5) After they've fixed it, the last step is to change the test
suite to expect success for that test, then commit.
6) Close the issue.
This is already easier than the system we have now, since merely
fixing the bug will cause an interesting change to the output of
"make check" (an expected failure will become an unexpected pass).
That's all we need.
But showing _all_ the unexpected failures with every test run is just
going to overwhelm everyone, because it presents an unorganized mass
of information from which no informed decision can be made. The issue
tracker presents the same information, but organized and prioritized,
making it easier for people to decide what to work on.
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Thu Aug 22 18:32:08 2002