Draft: r300: fail linking for some shaders where we know we will hit the instruction limit
In the past we were happily linking shaders that had zero chance to run and than we replaced them with dummy ones (for fs) or we skipped the draws altogether for vs in a hope that maybe the shader is not used at all or if it is used that maybe it will results in not so bad misrendering. Also not all apps (especially the old ones) handle linking failures well, so this was used to prevent them crashing and allow them to maybe display something.
However, some apps actually need a reliable info whether linking succeds so a fallback paths could be used instead. Most notably nowadays, glamor has a fallback sw rendering for gradient shaders, which we have no chance to run on R3xx in HW. So we want to fail liking there.
This implementation is kind of different from i915 !25533 (merged) where it really goes all the way and does a full compile in finalize_nir. The reason is that the r300 backend compiler is quite heavy and having to compile every shader twice increases loading times a lot and also runtime of stuff like dEQP and piglit by as much as 50%. So instead we try to estimate the final instruction count from NIR and make a decision based on that. Its in no way perfect and in fact is quite optimistic, so some shaders which won't run will still pass the check. But this is good enough for glamor and the overhead is minimal.
Once I figure out how to actually cache the shaders so that we can skip the second compile at least for the state-independent ones, we can compile all the way through and get rid of this guesswork.
There is about 160 lost shaders with my shader-db on R3xx and 9 with R4xx. In all cases this is either lost vs because we failed to link the corresponding fs, or the other way around, so no functional change. No changes on R5xx.
First commit introduces a switch to go back to the old behavior, either with an env var or driconf override...
This is draft right now, because I don't haven't done any actual testing besides shader-db, so I'm looking for some testers with R3xx to confirm glamor works with this assuming you have also xserver/xwayland with xorg/xserver!1173 (merged)