multi-deqp test plan support
In CI we need to run a bunch of little deqp test sets ("test with UBO opts disabled. Test with bypass forced. Test with sysmem forced. etc.") and running deqp-runner separately for each of those makes for verbose logs with the summaries scattered throughout, and wastes CPU time finishing off test groups when we could be running jobs from the next test set in parallel. Relatedly, we don't particularly care to have the CI jobs separated into GL vs GLES2 vs GLES31, and could merge them together.
My thought is to specify a .toml file or something with basically a set of deqp-runner args per deqp configuration. You would describe all the tests you want to run, the caselists and output dirs and baselines and fractions and etc. for them. Then specify the global fraction/fraction_start for that board, and have it run the appropriate fraction of everything you want to test.
If we do this well, we could probably save 5-10% of test runtime for cheza.
Some of the steps:
-
Deqp
(which references the binary and some of the options) needs to be associated with each test group, not the the whole parallel test invocation (done) - Summary printing can't be part of
Deqp
(done) - The top-level result collection can't have the test list (since separate
Deqp
s might have the same test names) - How do we distinguish which
Deqp
a failure is for in test logs and printed summary? - Multi-level fraction/fraction_start handling (within a `Deqp`` and global)