Shader-db: A collection of shaders for offline analysis

Shader-db is a giant pile of shaders from various apps, for whatever purpose. In particular, we use it to capture assembly output of the shader compiler for analysis of regressions in compiler behavior.

Currently it supports Mesa's i965 and radeonsi drivers.

Capturing shaders

Create the directory for shader files

$ mkdir dirpath

Write all shaders that are loaded by "executable" to the directory

$ MESA_SHADER_CAPTURE_PATH=<dirpath> <executable>

Use fdupes can be used to remove duplicates

$ fdupes -d <dirpath>

Compiling

Some libraries are required when building. See section "Dependencies" below. To build the binary, do: make

i965/iris Usage

Running shaders

$ ./run shaders 2> err | tee new-run

To run just a subset:

$ ./run shaders/supertuxkart 2> err | tee new-run

Running for another platform, use intel_stub_gpu (from Mesa repository) :

$ intel_stub_gpu -p icl ./run shaders 2> err | tee new-run

Make sure to check the contents of 'err' after your run.

To compile shaders for an i965 PCI ID different from your system, pass

-p {i965,g4x,ilk,snb,ivb,hsw,byt,bdw}

to run.

Analysis

$ ./report.py old-run new-run

radeonsi Usage

Running shaders

$ ./run shaders > new-run 2> /dev/null

Note that a debug mesa build required (ie. --enable-debug)

Analysis

./si-report.py old-run new-run

freedreno, v3d Usage

Running shaders

$ ./run shaders > new-run

Note that a debug mesa build required (ie. --enable-debug)

Analysis

./report.py old-run new-run

Dependencies

run requires some GNU C extensions, render nodes (/dev/dri/renderD128), libepoxy, OpenMP, and Mesa configured with --with-egl-platforms=x11,drm

jemalloc

Since run compiles shaders in different threads, malloc/free locking overhead from inside Mesa can be expensive. Preloading jemalloc can cut significant amounts of time:

$ LD_PRELOAD=/usr/lib64/libjemalloc.so.1 ./run shaders 2> err | tee new-run

Deprecated

run.py is obsolete. Use the run binary instead.

Vulkan Fossils

shader-db also contains a selection of vulkan fossils. These are generated using the fossilize.sh script and fossil.

You will need a vulkan driver with pipeline statistics:

$ fossilize-replay --enable-pipeline-stats output.csv --num-threads 4 fossils/**/*.foz

Or you can use a helper script to do this:

$ ./fossil_replay.sh fossils/ output.csv --num-threads 4

Or you can use fossil_prune_replay.sh to replay a specific pipeline by hash:

$ ./fossil_prune_replay.sh fossils/ 012897d03e30f26f

If you do not get a .csv file it likely means that a driver without VK_KHR_pipeline_statistics, such as a system installed driver.

You can then compare two different csv files using the report-fossil.py script:

$ report-fossil.py baseline.csv development.csv

Capturing fossils

A fossilize.sh script is provided to assist in capturing fossils, you may need to modify it based on where fossil lives on your system.

For steam games, fossils are currently captured automatically. Once the game has been run for a while, you can find the fossils at $HOME/.local/share/Steam/steamapps/shadercache/$APP_ID/fozpipelinesv4/steamapp_pipeline_cache.foz. Note that you need to use the file named steamapp_pipeline_cache.foz and not the fossil file named steam_* because the later one may contain fossilized pipelines from other drivers which may use extensions not supported by your driver. The steamapp_pipeline_cache.foz file contains only those pipelines which have actually been seen during the run on your actual driver.