Julian Foad wrote on Thu, 16 Jan 2020 09:18 +00:00:
> I inspected the debug prints whizzing by, and noted that it included
> each of the different kinds of input cases I expected (e.g. empty
> ranges, reversed ranges, duplicate ranges, overlapping ranges, etc.) and
> that the results included examples of the failure modes I expected.
>
> Based on that observation, I altered the parameters such as the sizes of
> the generated revision ranges and the repetition counts, until it
> provided good coverage without excessive repetition. I aimed for "a few"
> (more than one, less than hundreds) of occurrences of the same failure mode.
>
> That doesn't guarantee coverage of every possible edge case combination,
> of course, but it gives me confidence that it's an effective test. I
> admit it's an issue that that is not apparent to an observer. However,
> anyone needing to debug again can enable the debug prints and see it.
Julian and I discussed this on IRC today; the conclusion is:
- With the behaviour of the test with seed=0 having been manually
reviewed, we're satisfied that the test achieves adequate coverage.
- The properties that the magic numbers were chosen to satisfy will be
documented in code comments.
- Julian has further reviewed the code, leading to, e.g., r1872919.
Thanks, Julian.
Daniel
Received on 2020-01-17 16:36:12 CET