Daniel Shahaf wrote:
> Julian Foad wrote on Tue, Jan 07, 2020 at 09:47:48 +0000:
>> For now, I propose to make each test use a repeatable sequence, independent
>> of the other tests. I think that will be enough; the options I mentioned
>> can be added if and when there is a demand for them.
>
> I don't understand why you're fixing the seed.
>
> For repeatability, all you need is to output the seed to stdout and to be able
> to re-run the tests with a particular value of the seed:
The suggestion was that we should prefer the regression test suite to be
deterministic, running the same fixed set of tests each time. Repeating
by default, not just repeatable on request.
The other way (non-repeating by default, and repeatable on request) is a
possible alternative that we could choose. Indeed, a few (six or so)
other tests in the suite are currently written that way. However we
don't really have the test infrastructure for handling that nicely. We
could add some, starting with the pseudocode you posted, but we haven't
yet and it would probably want a bit more than that to make an overall
good experience.
> Conversely, the effect of fixing the seed is to make every single test run test
> exactly the same sequence of API calls on the same sequence of inputs. That
> means the test is not merely repeatable, but also deterministic
Yes, like nearly all of our test suite.
> — which begs
> the question, why do we need a PRNG at all? What do we gain
We gain the ability to test a large, diverse set of combinations of
inputs. Call it an "test case generator" instead of a "PRNG", if that
helps.
> in exchange for the
> costs and risks associated with using a PRNG in the test suite?
What?
- Julian
Received on 2020-01-08 11:14:44 CET