Ben Reser wrote:
> 1) Decrease the amount of testing done by the every commit buildbots on the
> branches. This bots should be a quick sanity check, does it build and does it
> pass basic tests.
>
> 2) Create a buildbot setup that generates a tarball on each branch nightly (if
> there were any changes). It would do this in the same process that a RM would
> use (and eventually would become the process for producing the tarball).
>
> 3) This tarball would then have various build bots run tests off it. These
> tests would be much more extensive. In particular it would run cross version
> (SERVER_MINOR_VERSION), packing, sharding and other variations of our test
> suite that frankly probably don't get as much attention now as they should.
>
> 4) Eventually make it so that the RM asks the buildbot to produce the release
> tarball, including running the nightly buildbots against it. RM then downloads
> the tarball, compares it to the branch_at_revision for differences looking for any
> unexpected diffs, and then signs it. It gets put up for everyone to approve.
>
> 5) Humans would no longer do much testing. We'd have buildbots for the major
> platforms, which would have been included by our manual testing from the past.
> The build bots would be running far more extensive tests than we ever would be
> likely to run on our own. So the only testing we'd need to do is verifying
> that the builds match the source we expect and whatever sanity testing we felt
> like we wanted to do.
One more thing we'd need to to before signing, of course: verify that the buildbot test runs have completed to our satisfaction (right tarball, sufficient platforms, etc.).
> This should lower the bar dramatically for getting tests/signatures out.
> Humans don't need to do this boring and repetitive work. It should let
> everyone spend more time on more important issues than manual testing.
>
> Infra had offered us some more buildbot resources here recently. This seems
> like the logical way to use them.
+1 to all you said.
- Julian
Received on 2013-08-30 10:38:38 CEST