On Sat, Nov 30, 2013 at 2:23 PM, Daniel Shahaf <d.s_at_daniel.shahaf.name>wrote:
> Daniel Shahaf wrote on Sat, Nov 30, 2013 at 15:19:56 +0200:
> > Stefan Fuhrmann wrote on Sat, Nov 30, 2013 at 08:48:30 +0100:
> > > Again, you are missing the point. You are saying that there
> > > are lots of possible configurations and various combinations.
> > > I'm saying that we *are* actually able to test many variants.
> > > One simply needs to update their overall test script.
> > There are classes of problems that our test suite simply isn't designed
> > to catch.
> > For example:
> > - We won't catch 'premature kill -9' bugs (r1494298)
> > - We won't catch concurrency bugs (r1302613)
> As another example, we have architectue-dependent code, for example
> the definition of SVN_UNALIGNED_ACCESS_IS_OK. In those cases, do we
> test the "unlikely" branch (eg: SVN_UNALIGNED_ACCESS_IS_OK=0) too?
Well, that is exactly the kind of things that a test script
should cover. Running just another combination of config
options add less than 2:30 total (or 2 minutes with ra_local).
So, being able to quickly run a great number of variants
is clearly useful here.
> My build script defines -DSVN_UNALIGNED_ACCESS_IS_OK=0 for precisely
> this reason.
I think we should make a list of combinations that should
be part of release testing. It is certainly a waste to do
[local, serf, svn] x [bdb, fsfs, fsx] x [1.0 .. 1.9] x [P(config options)]
Instead, maybe along the lines of:
[local, serf, svn] x [bdb, fsfs, fsx] x [1.9] x [config defaults]
[local, serf or svn] x [fsfs] x [1.9] x [Selected config option sets]
[local, serf or svn] x [fsfs] x [1.1 .. 1.9] x [config defaults]
[local, serf or svn] x [bdb] x [1.0] x [config defaults]
For the ra layer bit, I feel that having "all in one" and
"separate c/s" may trigger different failure scenarios.
BDB is deprecated and FSX is experimental, so let's
focus on FSFS for the non-standard setups. 1.9 should
cover the most code making it a good test bed for
config option variation. At least the standard config
should work with all format versions.
Makes sense? What combinations of config options
should we test?
Received on 2013-11-30 15:30:31 CET