ACO/NIR: Stack overflow compiling certain shaders on proton/wine where stack size is small
On recent versions of proton/wine, vkCreate*Pipelines() calls go through a so-called syscall handler, which limits the stack size to a pretty small value. I'm not sure exactly how large it is, but it should be around 512KB. This causes ACO to overflow the stack when compiling certain shaders due to recursive calls to nir_unsigned_upper_bound() with the following backtrace:
#0 0x00007ffff73b9737 in nir_unsigned_upper_bound () from /home/oschowa/builds/mesa/lib/libvulkan_radeon.so
#1 0x00007ffff73b98c2 in nir_unsigned_upper_bound () from /home/oschowa/builds/mesa/lib/libvulkan_radeon.so
#2 0x00007ffff73b98c2 in nir_unsigned_upper_bound () from /home/oschowa/builds/mesa/lib/libvulkan_radeon.so
#3 0x00007ffff73b98c2 in nir_unsigned_upper_bound () from /home/oschowa/builds/mesa/lib/libvulkan_radeon.so
#4 0x00007ffff73b9d02 in nir_unsigned_upper_bound () from /home/oschowa/builds/mesa/lib/libvulkan_radeon.so
#5 0x00007ffff73b9d02 in nir_unsigned_upper_bound () from /home/oschowa/builds/mesa/lib/libvulkan_radeon.so
#6 0x00007ffff73b98c2 in nir_unsigned_upper_bound () from /home/oschowa/builds/mesa/lib/libvulkan_radeon.so
...
#70 0x00007ffff73ba452 in nir_unsigned_upper_bound () from /home/oschowa/builds/mesa/lib/libvulkan_radeon.so
#71 0x00007ffff7514b3e in aco::(anonymous namespace)::visit_alu_instr(aco::isel_context*, nir_alu_instr*) [clone .lto_priv.0] () from /home/oschowa/builds/mesa/lib/libvulkan_radeon.so
#72 0x00007ffff7674b3b in aco::(anonymous namespace)::visit_cf_list(aco::isel_context*, exec_list*) [clone .isra.0] () from /home/oschowa/builds/mesa/lib/libvulkan_radeon.so
#73 0x00007ffff75670a7 in aco::(anonymous namespace)::visit_if(aco::isel_context*, nir_if*) [clone .lto_priv.0] () from /home/oschowa/builds/mesa/lib/libvulkan_radeon.so
#74 0x00007ffff7674eab in aco::(anonymous namespace)::visit_cf_list(aco::isel_context*, exec_list*) [clone .isra.0] () from /home/oschowa/builds/mesa/lib/libvulkan_radeon.so
#75 0x00007ffff75670a7 in aco::(anonymous namespace)::visit_if(aco::isel_context*, nir_if*) [clone .lto_priv.0] () from /home/oschowa/builds/mesa/lib/libvulkan_radeon.so
Note also that the syscall handler also causes this stack overflow to not crash the application, but rather the vkCreate*Pipelines() call just fails and the game keeps running.
It can be reproduced by replaying the following fossilize archive with a compute shader from the game "Stranger of Paradise: Final Fantasy Origin" when limiting the stack size to 512KB with ulimit -s 512
:
stack.foz
This archive compiles fine on ACO with at least 1MB of stack size, but both amdvlk, amdgpu-pro and NVIDIA manage to compile this shader with just 128KB of stack.