flake detection mode for the initial CI expectation updates
While writing a previous issue, I thought about the flakes and their detection.
So far, our workflow is to trigger the CI job multiple times and then gather the flake results and continuously update it.
What I think would be somewhat useful is to have the ability to override the default number of invocation of each test, so instead of retrying ONLY if the test fails, retry every time, let's say x times (3x/5x?), in case the tests gets flaky.
Real-world usage could be:
- Mesa3D CI nightly full job (takes 3hr)
- before we introduce the job to the MesaCI, we would run the job once (9hrs - 15hrs) with this option and write down all flakes
From script or operator perspective, it seems like a much more robust solution, where deqp-runner would do just the workflow it does when hits the failure.
edit: Possible cons:
- this workflow assumes stable runner, kernel, and drm. In case of unstability underneath it may be sad to see job crashes after 7 hours.