Re: [PATCH] input_validation_tests.py: Reduce overhead of multiple checkouts.

Stefan Sperling <stsp_at_elego.de> writes:

> On Mon, Feb 21, 2011 at 01:44:35PM +0530, Noorul Islam K M wrote:
>>
>> This patch reduces checkout by around 23 times.
>
> On my system the difference is 43 seconds vs. 30 seconds.

On my low-end Linux desktop it's 7.5 seconds and 3.5 seconds, run
sequentially on a SATA disk.

> We lose the ability to easily spot which of the subtest is failing
> if we do this. I.e. instead of:
>
> ...
> PASS: input_validation_tests.py 19: non-working copy paths for 'status'
> FAIL: input_validation_tests.py 20: non-working copy paths for 'patch'
> PASS: input_validation_tests.py 21: non-working copy paths for 'switch'
> ...
>
> all we'd get is:
>
> FAIL: input_validation_tests.py 1: inavlid wc and url targets

When a test fails the first thing I do is look in tests.log, that will
still work just as well with the combined test. I suppose we do lose
out if there are multiple independent bugs, as the test will stop at the
first one and not report the others.

I feel we should be optimising for the common case where the tests PASS,
and that optimising for FAILs is the wrong approach.

I agree that combining big, complex tests, like merge, is a bad idea.
But for relatively trivial tests I think reduced runtime is more
important.

-- 
Philip