Skip to content

ci-collate artifacts path

The location of the files in the jobs can have different paths depending on the job within the same pipeline.

The old example is in virglrenderer where results/failures.csv is the default location, but there are jobs (the sharded ones) that saves this file (among others in) results/{{ unsharded job name }}/failures.csv.

Newer example is mesa's jobs like zink-radv-vangogh-valve where they can have multiple failures.csv in different locations (up to three) following the structure job_folder/results/{gl,gles,piglit}/failures.csv.

When someone likes to collect the failures.csv file (or others like results.csv) from a pipeline, this must not be an obstacle.

The solution applied to ci-uprev is to get_artifact("*/failures.csv") and with this star at the beginning we drive the software to search in the artifacts for all the files with this name, then return the content like if it is a single file.

In ci-collate we need to do something similar:

  • When there is only one file with this name, return the Artifact() object and the path is stored in the attribute of the object.
  • When there is more than one, we may have to return a list of artifacts with each Artifact() object (having also the path stored).

There is a "room for improvement" in the solution for ci-uprev. It downloads the artifacts.zip to walk the paths looking for the files. This means to download many useless data, and it can have a size that impacts performance.

The current idea to solve that is to modify the CI (mesa, but also virgl and drm-ci) to delegate to each job to generate a json file with information about the content structure in the artifacts. Since then, this seems to be the best solution even the issue.