[Big Endian] [r300] vdpau output causes segmentation faults with bit depths other than 24 or 30bpp
Attempting to use vdpau video output with r300 and bit depths other than 24/30bpp on big endian results in a segmentation fault. According to !20524 (merged), segfaults occur under little endian systems too, but I have not yet tested if both segfaults are related.
While I was unable to bisect in order to find exactly which commit these crashes were introduced because of long ago it was introduced, I did do a bit of manual debugging.
Segmentation Fault reports occurring here: https://gitlab.freedesktop.org/mesa/mesa/-/blob/mesa-22.3.2/src/gallium/drivers/r300/r300_emit.c#L56
Edit: vl_dri2_format_for_depth
here returns PIPE_FORMAT_NONE
on non-24/30bpp depths, propagating down to the line above and causing a segfault.
Values returned by r300_surface(cb)->colormask_swizzle
appear to be the cause of segmentation faults here, perhaps size
is smaller than expected value for returned swz
and causing overflows.
That portion of code does not seem to have changed much, because of that, these crashes may have been introduced in 1e51d368 itself
Commenting out that line does make it run without segfaulting, although with incorrect/garbled output.
Running Xorg at 24bpp or 30bpp avoids segfaulting
CC @ondracka