Daniel and I had a chat about improving our release process today and I'd like
to make some proposals.
1) Decrease the amount of testing done by the every commit buildbots on the
branches. This bots should be a quick sanity check, does it build and does it
pass basic tests.
2) Create a buildbot setup that generates a tarball on each branch nightly (if
there were any changes). It would do this in the same process that a RM would
use (and eventually would become the process for producing the tarball).
3) This tarball would then have various build bots run tests off it. These
tests would be much more extensive. In particular it would run cross version
(SERVER_MINOR_VERSION), packing, sharding and other variations of our test
suite that frankly probably don't get as much attention now as they should.
4) Eventually make it so that the RM asks the buildbot to produce the release
tarball, including running the nightly buildbots against it. RM then downloads
the tarball, compares it to the branch_at_revision for differences looking for any
unexpected diffs, and then signs it. It gets put up for everyone to approve.
5) Humans would no longer do much testing. We'd have buildbots for the major
platforms, which would have been included by our manual testing from the past.
The build bots would be running far more extensive tests than we ever would be
likely to run on our own. So the only testing we'd need to do is verifying
that the builds match the source we expect and whatever sanity testing we felt
like we wanted to do.
This should lower the bar dramatically for getting tests/signatures out.
Humans don't need to do this boring and repetitive work. It should let
everyone spend more time on more important issues than manual testing.
Infra had offered us some more buildbot resources here recently. This seems
like the logical way to use them.
Received on 2013-08-29 23:48:30 CEST