Julian Foad wrote on Wed, Jan 08, 2020 at 10:14:42 +0000:
> Daniel Shahaf wrote:
> > Julian Foad wrote on Tue, Jan 07, 2020 at 09:47:48 +0000:
> > > For now, I propose to make each test use a repeatable sequence, independent
> > > of the other tests. I think that will be enough; the options I mentioned
> > > can be added if and when there is a demand for them.
> >
> > I don't understand why you're fixing the seed.
> >
> > For repeatability, all you need is to output the seed to stdout and to be able
> > to re-run the tests with a particular value of the seed:
>
> The suggestion was that we should prefer the regression test suite to be
> deterministic, running the same fixed set of tests each time. Repeating by
> default, not just repeatable on request.
>
> The other way (non-repeating by default, and repeatable on request) is a
> possible alternative that we could choose. Indeed, a few (six or so) other
> tests in the suite are currently written that way. However we don't really
> have the test infrastructure for handling that nicely. We could add some,
> starting with the pseudocode you posted, but we haven't yet and it would
> probably want a bit more than that to make an overall good experience.
Makes sense.
> > Conversely, the effect of fixing the seed is to make every single test run test
> > exactly the same sequence of API calls on the same sequence of inputs. That
> > means the test is not merely repeatable, but also deterministic
>
> Yes, like nearly all of our test suite.
But unlike nearly all of the test suite, we don't actually know what the
sequence of inputs _is_, and we have no reason to believe it provides good
coverage of edge cases and rare combinations of inputs. For all we know, the
90000Â iterations could all be testing the exact same, most common codepath over
and over again.
> > — which begs
> > the question, why do we need a PRNG at all? What do we gain
>
> We gain the ability to test a large, diverse set of combinations of inputs.
> Call it an "test case generator" instead of a "PRNG", if that helps.
>
> > in exchange for the
> > costs and risks associated with using a PRNG in the test suite?
>
> What?
Using a PRNG makes the test code harder to read and to maintain, makes it
more difficult to diagnose FAILs, and increases the risk of a bug in the test
code. What benefits are we getting that offset these risks and costs?
The way in which the PRNG algorithm is used is outside the PRNG's API promises.
PRNG's make *no* promises on their outputs when used with a fixed seed. In
light of this, what reason do we have to believe that the test achieves good coverage?
It's not necessary to use a PRNG to test a large/diverse set of inputs. You
can achieve with plain old combinatorics (e.g., define 300 rangelist variables
and test various combinations of them). I'd actually expect that to have
better coverage than a PRNG with a fixed seed.
By default, I would have expected the tests to be written without a PRNG, and I
don't understand why the way the tests are is better.
Cheers,
Daniel
Received on 2020-01-08 15:24:53 CET