Skip to content

Add experimental support for skqp

Helen Mae Koike Fornazier requested to merge helen.fornazier/deqp-runner:skqp into main

skqp stands for SKIA Quality Program conformance tests. Basically, it has sets of rendering tests and unit tests to ensure that SKIA is meeting its design specifications on a specific device.

Add skqp-runner binary to execute skqp tests.

The threashold number is concatenated on the name of the test. Exemple: gl_3dgm@3, where 3 is the threadshold of the skqp test. This allows concatenating test results that ran the same tests with different threasholds, and also allows passing a baseline with different expected result per threashold.

For the tests which failed, the skqp output with html results is kept on the output folder for later analisis of the user.

Since skqp requires tests to be placed on $(--skqp-folder)/skqp/skqp/rendertests.txt $(--skqp-folder)/skqp/skqp/unittests.txt to retrieve threasholds. The files passed through --caselist-rendertests and --caselist-unittests are copied to that location before executing tests.

This is an example of end results, running on a mesa-ci tree: (check README for an example with skia tree)

root@biscuit:/# /skqp-runner run --skqp /skqp/skqp --output /tmp/result --skqp-folder /skqp/assets --skqp-backends gl --caselist-rendertests /skqp/assets/skqp/rendertests.txt
Running 2 skqp tests on 8 threads
Pass: 0, Duration: 0
ERROR - Skqp error: libGL error: MESA-LOADER: failed to retrieve device information
ERROR - Skqp error: libGL error: Version 4 or later of flush extension not found
ERROR - Skqp error: libGL error: failed to load driver: i915
ERROR - Skqp error: vkCreateInstance failed: -9
ERROR - Test gl_3dgm@-1: Crash: See "/tmp/result/skqp.gl_3dgm@-1.log" and "/tmp/result/gl_3dgm@-1.c1.r1"
ERROR - Skqp error: libGL error: MESA-LOADER: failed to retrieve device information
ERROR - Skqp error: libGL error: Version 4 or later of flush extension not found
ERROR - Skqp error: libGL error: failed to load driver: i915
ERROR - Skqp error: vkCreateInstance failed: -9
ERROR - Skqp error: libGL error: MESA-LOADER: failed to retrieve device information
ERROR - Skqp error: libGL error: Version 4 or later of flush extension not found
ERROR - Skqp error: libGL error: failed to load driver: i915
ERROR - Skqp error: vkCreateInstance failed: -9
ERROR - Test gl_3dgm@-1: Crash: See "/tmp/result/skqp.gl_3dgm@-1.log" and "/tmp/result/gl_3dgm@-1.c1.r2"
Pass: 1, Crash: 1, Duration: 0, Remaining: 0

Slowest tests:
  gl_CubicStroke@-1 (0.38s)
  gl_3dgm@-1 (0.29s)

Some failures found:
  gl_3dgm@-1,Crash
root@biscuit:/# ls -R /tmp/result
/tmp/result:
failures.csv  gl_3dgm@-1.c1.r1  gl_3dgm@-1.c1.r2  results.csv  skqp.gl_3dgm@-1.log

/tmp/result/gl_3dgm@-1.c1.r1:
grdump.txt  out.csv  report.html  unit_tests.txt

/tmp/result/gl_3dgm@-1.c1.r2:
grdump.txt  out.csv  report.html  unit_tests.txt

Signed-off-by: Helen Koike helen.koike@collabora.com

See #33 (closed)

Edited by Helen Mae Koike Fornazier

Merge request reports