Mark Phippard wrote:
> There were 11 release candidates. But 5 were never released because
> our own test suite failed and one had to be reissued two days later.
> In addition the last 4 release candidates came out within the last two
> weeks. This tells me a couple things:
>
> 1) We need to fix our process that leads up the RC so that test
> results do not come as a surprise. Ideally we could make better use
> of buildbot and have it do a complete test run on the branch on a
> regular basis. But maybe there are other options. The -preN releases
> are not going to help unless people are going to run all the tests.
> Given that there is nothing stopping us from doing it without these
> tarballs, I do not think releasing them is going to fix this. I still
> think they are a good idea, they just are not going to fix this alone.
>
> 2) We did not get any real community feedback until we were
> threatening to do the final release. I do not think there is much we
> can do about this, it is something that I think is just an unknown
> factor we always have to deal with. There are a lot of tools and
> scripts that use our libraries and we know our test suite is weak in
> this area. I expect this to be a continuing problem in future
> releases. That said, if our release process becomes more efficient,
> we may find that a lot of these things start getting reported and
> fixed in ".1" releases because we got the release out before community
> people started to really test it.
The TortoiseSVN nightly builds (trunk, not from the stable branch) are
always linked against the Subversion trunk (updated regularly, the
svn:externals are pulled in with -r specified to avoid build breakage).
There are a few people who download and use those. These people usually
update on a weekly basis (or sooner if they want to test a new feature
or check whether a bug they found might be fixed).
But even with those people using trunk, it's simply not possible to find
all issues. Especially features which they don't use regularly won't get
tested a lot.
With 1.5 the problem was for most of our testers that they couldn't test
against an 1.5 server. So merge tracking wasn't tested as well as we'd
hoped.
What also would help a lot is if there was an official test server
available - maybe with public write access, but the repo would get
purged every few hours automatically?
For example, the sasl authentication was hard to test because I couldn't
get a server working on my Windows machine. When users started to report
problems with it, I had no idea whether the issue was in the code or in
their setup - it we had a test server, I could just point them there: it
it works with the test server, it's their setup. If not, I know that
it's not a setup issue and can look in the code.
Stefan
--
___
oo // \\ "De Chelonian Mobile"
(_,\/ \_/ \ TortoiseSVN
\ \_/_\_/> The coolest Interface to (Sub)Version Control
/_/ \_\ http://tortoisesvn.net
Received on 2008-09-17 18:24:06 CEST