Skip to content

util/disk_cache: implement single file shader cache

Timothy Arceri requested to merge tarceri/mesa:refactor_cache into master

My benchmarking shows no significant change in cache load times with a single shader cache file vs the existing cache implementation of many small files (tested with my spinning rust HDD).

However this new single file cache implementation does reduce the total size on disk used by the shader cache. We have a problem with the existing cache where writing tiny files to disk causes more disk space to be used than is actually needed for the files due to the minimum size required for a file. In pratice this tends to inflate the size of the cache on disk to over 3x larger.

There are other advantages of using a single file for shader cache entries such as allowing better removal of cache entries once we hit the max cache size limit (although we don't implement any max cache size handling in this initial implementation).

The primary reason for implementing a single file cache for now is to allow better performance and handling by third party applications such as steam that collect and distribute precompiled cache entries.

For this reason we also implement a new environment variable MESA_DISK_CACHE_READ_ONLY_FOZ_DBS which allows a user to pass in a path to a number of external read only shader cache dbs. There is an initial limit of 8 dbs that can be passed to mesa like so:

MESA_DISK_CACHE_READ_ONLY_FOZ_DBS=/full_path/filename1, ... ,/full_path/filename8

Where the filename represents the cache db and its index file e.g. filename1.foz and filename1_idx.foz

For now the single file cache implementation is optional and the original cache implementation remains default. When the MESA_DISK_CACHE_SINGLE_FILE environment variable is set we make use of the new single file shader cache implementation.

The new cache uses the following directory structure based on the first defined name as follows:

$MESA_GLSL_CACHE_DIR/driver_id/gpu_name/foz_cache.foz $MESA_GLSL_CACHE_DIR/driver_id/gpu_name/foz_cache_idx.foz

$XDG_CACHE_HOME/mesa_shader_cache_sf/driver_id/gpu_name/foz_cache.foz $XDG_CACHE_HOME/mesa_shader_cache_sf/driver_id/gpu_name/foz_cache_idx.foz

<pwd.pw_dir>/.cache/mesa_shader_cache_sf/driver_id/gpu_name/foz_cache.foz <pwd.pw_dir>/.cache/mesa_shader_cache_sf/driver_id/gpu_name/foz_cache_idx.foz

Where foz_cache_idx.foz is a database of offsets pointing to the location of the shader cache entries in foz_cache.foz

This initial implementation doesn't have any max cache size handling and is initially intended to be use by applications such as steam that will handle cache management for us.

Edited by Timothy Arceri

Merge request reports