"Jostein Chr. Andersen" <jostein@josander.net> writes:
> If we look at we have called RC's so far, then the RC1 was of beta
> quality IMHO. So according to the old standards: reset the soak time -
> again IMHO.
(Fortunately, we're at leisure to discuss a soak time reduction even
as the soak time clock continues ticking!)
I'm somewhat uncomfortable with a 3 week soak for RC2. The real issue
is not the "beta" quality of RC1, but the amount of code churn on the
branch between RC1 and RC2, and the risk that some of that churn
introduced bugs that were not present in RC1. I'm willing to be
persuaded that 3 weeks is enough, especially with added intentional
testing, but... Well, read on.
During the week or so of RC1 soak, several bugs were found. At least
one of these was a fairly major bug on Windows, related the new
locking code -- under certain circumstances, it broke 'svn update'.
These circumstances were not a complete edge case, but they wouldn't
come up in most people's daily use of Subversion either. Thus,
although we might expect that such a bug on Windows would be caught by
lots and lots of people, IIRC only one user noticed it (or anyway, one
user reported it). Fortunately, Moisei was able to give us a good
reproduction recipe, so it was fixed in r14304.
That whole episode really made me think: it would have sucked to ship
with that bug, yet it was caught by just *one* user? True, it
happened early in the soak period, so maybe over a full four week
soak, others would have reported it too. But still...
I'm not -1'ing the 3-week proposal. Yet. I've started reviewing all
the merges onto the 1.2.x line between RC1 and RC2. I'd rather base
such a decision on a close examination of the changes in question,
than on vague, fuzzy feelings about code churn. If in the end I'm
still jittery about a shortened soak, and want to propose a full soak,
I'll at least have details to back it up. (One thing I wish we had
stats for is, how many -- and what sort of -- bugs were found in 1.1.0
during its final week of soak.) In the meantime, the clock is still
ticking, so we're not losing any ground.
On a more general note:
Subversion chose a 4 week soak time somewhat arbitrarily. Does
reducing it to 3 weeks reduce its effectiveness by one quarter?
Probably not, because bugs found toward the end of the soak are more
likely to be duplicates of bugs found earlier in the soak anyway.
Going from 4 to 3 reduces the soak's effectiveness by *some* amount,
but we really don't know how much.
My feeling is, if we're comfortable with a 3 week soak for RC2 --
given additional testing -- then we ought to consider a more general
policy change. Such as:
The total soak time of all release candidates in a minor line must
be at least 4 weeks. The last release candidate must have soaked
for at least 2 weeks before being promoted to a final release.
Thoughts?
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Thu Apr 28 23:32:22 2005