Automatically update expectations based on the Nightly runs
We are daily running all the tests in the CI using the scheduled pipeline popularly known as nightly runs. Those results have a huge value but we usually have to gain this knowledge by hand. One of those results that can be automatically processed is the changes in the expectations.
I did one attempt some time ago ( !27461 (closed)) and I launched a second one today (!28350 (merged)).
Currently, ci-collate
(See gl.fd.o/gfx-ci/ci-collate) has a subcommand called patch
(See gfx-ci/ci-collate!15 (merged)) but it needs some refinement to be called automatically:
The list of ideas start here:
-
Would this be triggered because a nightly run has finished? -
Generate a MR (Would it be automatically assigned to marge? (When this has been tested enough, not too early)) -
Header Nightly run expectations update
when inserting a block of tests (related with gfx-ci/ci-collate#6)-
Second line on the header with a compare url (See !27461 (comment 2268741))
-
-
Do not remove tests from flakes (insert allowed) (See !27461 (comment 2281779)) -
Allow to use the results of many Nightly runs, to then have enough data to think about flake removals (related with gfx-ci/ci-collate#9)
-
As feedback is received, this list will be refined.
Edited by Sergi Blanch Torné