Julian, would you like to answer the question in the second quoted paragraph?
Daniel Shahaf wrote on Wed, Jan 08, 2020 at 14:24:39 +0000:
> Julian Foad wrote on Wed, Jan 08, 2020 at 10:14:42 +0000:
> > What?
>
> Using a PRNG makes the test code harder to read and to maintain, makes it
> more difficult to diagnose FAILs, and increases the risk of a bug in the test
> code. What benefits are we getting that offset these risks and costs?
>
> The way in which the PRNG algorithm is used is outside the PRNG's API promises.
> PRNG's make *no* promises on their outputs when used with a fixed seed. In
> light of this, what reason do we have to believe that the test achieves good coverage?
>
> It's not necessary to use a PRNG to test a large/diverse set of inputs. You
> can achieve with plain old combinatorics (e.g., define 300 rangelist variables
> and test various combinations of them). I'd actually expect that to have
> better coverage than a PRNG with a fixed seed.
>
> By default, I would have expected the tests to be written without a PRNG, and I
> don't understand why the way the tests are is better.
>
> Cheers,
>
> Daniel
Received on 2020-01-15 18:50:09 CET