Add test-suite support for deqp-runner.
Lets you specify a toml file to run multiple deqps from the same runner for better CPU usage and results visualization:
[[deqp]]
deqp = "/home/anholt/src/VK-GL-CTS-build/modules/gles2/deqp-gles2"
caselist = ["/home/anholt/src/VK-GL-CTS-build/external/openglcts/modules/gl_cts/data/mustpass/gles/aosp_mustpass/3.2.5.x/gles2-master.txt"]
baseline = "/home/anholt/src/mesa/src/gallium/drivers/softpipe/ci/deqp-softpipe-fails.txt"
skips = ["/home/anholt/src/mesa/.gitlab-ci/deqp-all-skips.txt", "/home/anholt/src/mesa/src/gallium/drivers/softpipe/ci/deqp-softpipe-skips.txt"]
deqp_args = [
"--deqp-surface-width=256",
"--deqp-surface-height=256",
"--deqp-surface-type=pbuffer",
"--deqp-gl-config-name=rgba8888d24s8ms0",
"--deqp-visibility=hidden"
]
[[deqp]]
deqp = "/home/anholt/src/VK-GL-CTS-build/modules/gles3/deqp-gles3"
caselist = ["/home/anholt/src/VK-GL-CTS-build/external/openglcts/modules/gl_cts/data/mustpass/gles/aosp_mustpass/3.2.5.x/gles3-master.txt"]
baseline = "/home/anholt/src/mesa/src/gallium/drivers/softpipe/ci/deqp-softpipe-fails.txt"
skips = ["/home/anholt/src/mesa/.gitlab-ci/deqp-all-skips.txt", "/home/anholt/src/mesa/src/gallium/drivers/softpipe/ci/deqp-softpipe-skips.txt"]
deqp_args = [
"--deqp-surface-width=256",
"--deqp-surface-height=256",
"--deqp-surface-type=pbuffer",
"--deqp-gl-config-name=rgba8888d24s8ms0",
"--deqp-visibility=hidden"
]
deqp-runner suite --output output/ --testlog-to-xml ~/src/VK-GL-CTS-build/executor/testlog-to-xml --suite ~/cts-softpipe.toml
TODO:
- Do we like the command line interface?
- Are the keys in the right places?
- Should baseline or skips be top-level? Should we do it at both levels like fractions?
- should output dir be configurable per test?
- Add more integration tests?
- Decide if we want to integrate with piglit at all.
- Add test prefixes so we can distinguish the same testcase when run with different config knobs
Edited by Emma Anholt
Merge request reports
Activity
Filter activity
added 4 commits
Toggle commit listenabled an automatic merge when the pipeline for e8bef3b5 succeeds
I like this.
Some very late feedback, based on your TODO list...
TODOS:
- Are the keys in the right places?
- Should baseline or skips be top-level? Should we do it at both levels like fractions?
- [chadv] The example toml shows the expectation file keys at the per-exe level. That feels like the right approach.
- should output dir be configurable per test?
- [chadv] What did you decide here? (I haven't tried the new toml feature yet). It feels like we need some way of differentiating the outputs from different [[deqp]] blocks when the multiple dEQP blocks may run the same test with different params. Maybe output the results into different output dirs, named with toml key deqp.prefix; or prefix the output files with key deqp.prefix; or something else?
- Should baseline or skips be top-level? Should we do it at both levels like fractions?
- Add test prefixes so we can distinguish the same testcase when run with different config knobs
- [chadv] Yes. A toml key for a descriptive prefix is a great idea.
Edited by Lina Versace- Are the keys in the right places?
Please register or sign in to reply