C. Michael Pilato wrote:
> I suspect that merge tracking testing is no more complex than any other
> test, but *unlike* most of our other testing, it involves arbitrarily
> large and/or deep-historied datasets. Most of our regression tests can
> do their job in under 5 revisions of change. Merge tracking tests
> almost certainly need a bigger and more branch-ful / history-interesting
> dataset to play with.
>
> Because of this, I unfortunately can't help but to suspect that the push
> for "real-world testing" comes partly out of a lack of willingness to
> invest the required energies into writing fully automated functional and
> regression tests for this feature. It's simply going to take a
> non-negligible amount of time to define and compose those tests, and
> that work isn't very glorious.
Next to the unit tests (both the ones in C as the python tests), the
merge-tracking feature requires data-driven tests, which test a small
number of use cases with a large number of different data sets. Our test
framework doesn't support this kind of tests very well, as setting up a
test repository even with a few revisions requires an overwhelmingly
large amount of lines of code.
> It would be "easier" just to do adhoc
> testing, where "easier" simply means that the testing cost is amortized
> across the life of the unreleased feature rather than paid upfront
> during automated test composition. But to get the best quality out of
> the thing, you need those automated regression tests *anyway*.
>
+1, we need both usage and dedicated testing before we can release 1.5.0
with merge-tracking. We'll never get most scenario's covered in time for
release by doing only one of the two.
For 1.4.0 I've made a list of all new features and fixes and added the
regressions tests where they weren't available yet. For 1.5.0 I'd like
to do the same, but this time I'll send that list to the dev group and
invite people to assist writing them. At the same time I can setup a
test 'framework' for the merge-tracking feature, preferably together
with someone else who already knows how it's supposed to work.
When is 1.5.0 scheduled for release?
Lieven
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Fri Dec 15 17:25:33 2006