On Sat, Nov 30, 2013 at 2:23 PM, Daniel Shahaf <d.s_at_daniel.shahaf.name>wrote:
> Daniel Shahaf wrote on Sat, Nov 30, 2013 at 15:19:56 +0200:
> > Stefan Fuhrmann wrote on Sat, Nov 30, 2013 at 08:48:30 +0100:
> > > Again, you are missing the point. You are saying that there
> > > are lots of possible configurations and various combinations.
> > > I'm saying that we *are* actually able to test many variants.
> > > One simply needs to update their overall test script.
> >
> > There are classes of problems that our test suite simply isn't designed
> > to catch.
> >
> > For example:
> >
> > - We won't catch 'premature kill -9' bugs (r1494298)
> > - We won't catch concurrency bugs (r1302613)
>
> As another example, we have architectue-dependent code, for example
> the definition of SVN_UNALIGNED_ACCESS_IS_OK. In those cases, do we
> test the "unlikely" branch (eg: SVN_UNALIGNED_ACCESS_IS_OK=0) too?
>
Well, that is exactly the kind of things that a test script
should cover. Running just another combination of config
options add less than 2:30 total (or 2 minutes with ra_local).
So, being able to quickly run a great number of variants
is clearly useful here.
> My build script defines -DSVN_UNALIGNED_ACCESS_IS_OK=0 for precisely
> this reason.
>
I think we should make a list of combinations that should
be part of release testing. It is certainly a waste to do
something like
[local, serf, svn] x [bdb, fsfs, fsx] x [1.0 .. 1.9] x [P(config options)]
Instead, maybe along the lines of:
[local, serf, svn] x [bdb, fsfs, fsx] x [1.9] x [config defaults]
[local, serf or svn] x [fsfs] x [1.9] x [Selected config option sets]
[local, serf or svn] x [fsfs] x [1.1 .. 1.9] x [config defaults]
[local, serf or svn] x [bdb] x [1.0] x [config defaults]
For the ra layer bit, I feel that having "all in one" and
"separate c/s" may trigger different failure scenarios.
BDB is deprecated and FSX is experimental, so let's
focus on FSFS for the non-standard setups. 1.9 should
cover the most code making it a good test bed for
config option variation. At least the standard config
should work with all format versions.
Makes sense? What combinations of config options
should we test?
-- Stefan^2.
Received on 2013-11-30 15:30:31 CET