> -----Original Message-----
> From: Julian Foad [mailto:julianfoad_at_btopenworld.com]
> Sent: vrijdag 30 augustus 2013 10:38
> To: Ben Reser
> Cc: Subversion Development
> Subject: Re: Improving our release process
>
> Ben Reser wrote:
> > 1) Decrease the amount of testing done by the every commit buildbots on
> the
> > branches. This bots should be a quick sanity check, does it build and
does it
> > pass basic tests.
So we only see problems a day later?
And can check our fixes the day after.
Maybe we should then also ask every committer to test on Windows and Unix
before every commit?
(Or maybe ask them to setup their own bots, because we can't provide them
the current test coverage)
Why reduce the testing on the existing buildbots?
We should just add 'fast bots' if we think the current bots are too slow. I
don't think new hardware was added at the primary bots for the last 2-3
years, so just spending some money there should improve the build times
without reducing the test coverage given within a few hours.
The openbsd bot is now broken for almost a week, and with a setup to only
test once per 24 hours more bots would be in this state for longer periods.
> > 2) Create a buildbot setup that generates a tarball on each branch
nightly (if
> > there were any changes). It would do this in the same process that a RM
> would
> > use (and eventually would become the process for producing the tarball).
Didn't we already have one?
See:
http://ci.apache.org/builders/svn-trunk-nightly/builds/875
> >
> > 3) This tarball would then have various build bots run tests off it.
These
> > tests would be much more extensive. In particular it would run cross
> version
> > (SERVER_MINOR_VERSION), packing, sharding and other variations of our
> test
> > suite that frankly probably don't get as much attention now as they
should.
I don't see what it would help to run the bots run on tarballs vs running it
on specific revisions.
It will certainly cost a lot of time to rewrite the scripts, while the
buildbot can just handle this for tags/branches/revisions.
I try to build from tags/revisions whenever possible, as this allows me to
always get the exact sourcecode for files back via the SCM information
stored in the debug information without having to keep all versions of all
tarballs.
I trust that our tags and revisions in Subversion don't change. Don't you?
> >
> > 4) Eventually make it so that the RM asks the buildbot to produce the
> release
> > tarball, including running the nightly buildbots against it. RM then
> downloads
> > the tarball, compares it to the branch_at_revision for differences looking
for
> any
> > unexpected diffs, and then signs it. It gets put up for everyone to
approve.
> >
> > 5) Humans would no longer do much testing. We'd have buildbots for the
> major
> > platforms, which would have been included by our manual testing from the
> past.
> > The build bots would be running far more extensive tests than we ever
> would be
> > likely to run on our own. So the only testing we'd need to do is
verifying
> > that the builds match the source we expect and whatever sanity testing
we
> felt
> > like we wanted to do.
>
> One more thing we'd need to to before signing, of course: verify that the
> buildbot test runs have completed to our satisfaction (right tarball,
sufficient
> platforms, etc.).
The manual testing also adds coverage outside the scripted tests. My scripts
make my tests run almost identical as on the buildbot, but over the last 5
years I found more than a few release problems in the difference between
that 'almost identical' and the buildbots itself.
> > This should lower the bar dramatically for getting tests/signatures out.
> > Humans don't need to do this boring and repetitive work. It should let
> > everyone spend more time on more important issues than manual testing.
> >
> > Infra had offered us some more buildbot resources here recently. This
> seems
> > like the logical way to use them.
>
> +1 to all you said.
>
> - Julian
Received on 2013-08-30 13:02:10 CEST