Skip to content

Introduce the expectations file

Guilherme Gallo requested to merge gallo/deqp-runner:add-expectations-file into main

Based on: !74 (closed)

Add an expectation.json file that contains the list of files used for flakes and skips as well as the baseline file.

When running a suite with a TOML file, the expectation for each entries of the suite are listed in the expectation file.

This is mainly used to ensure that a result directory can be easily matched with the correct expectation files that have be used for those results.

An optional name field is added in the SubConfig struct to allow finer matching when a suite contains a lot of entries.

Example:

deqp-runner suite --suite suite.toml --output /tmp/results

suite.toml contents:

[[skqp]]
skqp = "/skqp/out/skqp-x64/skqp"
skqp_assets = "/skqp/src/main/assets"
fraction = 100


[[deqp]]
deqp = "/VK-GL-CTS/build/external/openglcts/modules/glcts"
caselists = [
    "/VK-GL-CTS/external/openglcts/data/gl_cts/data/mustpass/gl/khronos_mustpass/main/gl46-main.txt",
    "/VK-GL-CTS/external/openglcts/data/gl_cts/data/mustpass/gl/khronos_mustpass/main/gl45-main.txt",
]
flakes = ["/tmp/flakes.txt"]
fraction = 100
renderer_check = "GT2"

It will produce the /tmp/results/expectation.json file:

[
  {
    "name": "dqp0",
    "baseline": null,
    "flakes": ["/tmp/flakes.txt"],
    "skips": []
  },
  {
    "name": "skqp0",
    "baseline": null,
    "flakes": [],
    "skips": []
  }
]

Merge request reports