On Sat, Nov 30, 2013 at 2:19 PM, Daniel Shahaf <d.s_at_daniel.shahaf.name>wrote:
> Stefan Fuhrmann wrote on Sat, Nov 30, 2013 at 08:48:30 +0100:
> > Again, you are missing the point. You are saying that there
> > are lots of possible configurations and various combinations.
> > I'm saying that we *are* actually able to test many variants.
> > One simply needs to update their overall test script.
>
> There are classes of problems that our test suite simply isn't designed
> to catch.
>
> For example:
>
> - We won't catch 'premature kill -9' bugs (r1494298)
> - We won't catch concurrency bugs (r1302613)
>
It is true that our test suite does not cover these scenarios.
This has always been the case and so far did not prevent
releases.
My private testing with larger repositories (a.o. & friends)
had me identify a concurrency / consistency issue with
'svnadmin pack' during a 'svnadmin load'
In the future, however, we can of course add tests that
either kill some test application or trigger some yet-to-be-
designed kill / suicide callbacks in the code. My recent
change to make the C tests execute in parallel is to some
extend intended to simply put "unusual" stress on our code.
And that identified a few points where our code or APR
did not behave as expected by me at least.
1.8 added the ability to generate code coverage reports
for our tests (not sure that currently works). So, we should
also be able to identify critical bits that don't get any
coverage.
> So, no, running "make check" under more configurations (despite still
> being an improvement) won't make me more confident that FSFS7 is in
> release-shape. It simply doesn't test enough code paths.
>
Well, Ivan's argument was along the lines of "many formats"
and their combination over the lifetime of a given repository.
That's a fair point and this one *can* be addressed be running
our test suite through many (all?) of these combinations.
BTW, the log-addressing branch is not the first incarnation
of that code. Developed in the format7 branch, it underwent
review (by me) and improvement while it got transferred
to the log-addressing branch. There it received stress testing
and further fixes. Finally, Philip gave it an in-depth review
with a specific focus on potential concurrency issues. A
few minor things showed up and got fixed.
Now, we are at a point were internal review is certainly
welcome and will probably find more issues but the real
test is using it in the real world, e.g. as part of an alpha
release.
-- Stefan^2.
Received on 2013-11-30 15:10:43 CET