ASTC texture decompression fails when using software fallback
The Mesa fallback ASTC decoder fails to decode certain ASTC textures from Animal Crossing: New Horizon running on the Ryujinx emulator.
Ryujinx attempts to detect hardware ASTC support by checking the available extensions, and looking for ASTC related ones (specifically KHR_texture_compression_astc_ldr
). Mesa seems to expose these on all hardware, irrespective of the underlying hardware's native support (eg. Intel iGPU have native ASTC support, but AMD GPUs do not, ASTC support is exposed for both).
In the case of GPUs that do not support ASTC natively, it appears Mesa uses a software fallback implementation of an ASTC decoder. This decoder has issues decoding textures from the game Animal Crossing: New Horizon running on the Ryujinx emulator. It will either decode junk (which can be seen in the below screenshot), or it attempts to allocate a massive buffer during decoding, and returns a GL_OUT_OF_MEMORY error.
A user has provided an apitrace on this message. Replaying of the trace will only reproduce the issue on devices that do not have native ASTC support. Replaying on an Intel iGPU for example, will not reproduce the issue, as Mesa will upload the compressed texture directly.
The offending code path appears to be this: https://gitlab.freedesktop.org/mesa/mesa/-/blob/master/src/mesa/state_tracker/st_cb_texture.c#L375
A screenshot of the emulator ignoring the Mesa ASTC support, and using it's internal ASTC decoder: