Re: [#4840] Merge assertion failure in svn_sort__array_insert

On Wed, Jan 8, 2020 at 9:24 AM Daniel Shahaf <d.s_at_daniel.shahaf.name> wrote:
> Julian Foad wrote on Wed, Jan 08, 2020 at 10:14:42 +0000:
> > The suggestion was that we should prefer the regression test suite to be
> > deterministic, running the same fixed set of tests each time. Repeating by
> > default, not just repeatable on request.

> But unlike nearly all of the test suite, we don't actually know what the
> sequence of inputs _is_, and we have no reason to believe it provides good
> coverage of edge cases and rare combinations of inputs. For all we know, the
> 90000 iterations could all be testing the exact same, most common codepath over
> and over again.

That occurred to me as well. There's no way to 'uniq' the values when
they're being determined in a transient manner.

My earlier thinking (though I didn't spell it out) was to move the
PRNG code to a separate program, use it to generate a self-documenting
C array of unique test cases, and put that C array in the test suite.
If someone wanted to regenerate the array, they could re-run the
program that generates them. If we discover new failing inputs in the
future, they can be added to the test suite. Then there is no guessing
about what inputs are being tested.

I haven't yet decided whether I still like this idea, because Daniel's
thoughts seem much more sensible:

> It's not necessary to use a PRNG to test a large/diverse set of inputs. You
> can achieve with plain old combinatorics (e.g., define 300 rangelist variables
> and test various combinations of them). I'd actually expect that to have
> better coverage than a PRNG with a fixed seed.

In other words, replace the PRNG and leave all else as-is.

Generating all permutations when picking 2 items is easy:

    for (i = 0; i < 299; i++) {
        for (j = i + 1; j < 300; j++) {
            test(i, j);
            test(j, i);
        }
    }

This generates all 89,700 permutations (300 nPr 2 = 300!/298!) and
there's no huge file of test cases to generate.

Nathan
Received on 2020-01-08 16:56:19 CET