Incorrect format conversion on big endian
I do believe I found a bug in some Mesa format conversions on big endian (Oracle Solaris 11.4 SPARC). The issue started with screenshots from gnome-screenshot having some channels swapped. I went through the stack trace ended up in Mesa and its _mesa_format_convert()
function. Here's what I found (but please know that I know almost nothing about Mesa, so some of my assumptions might be wrong):
In read_rgba_pixels(), _mesa_format_convert()
is in my case called with dst_format
being MESA_FORMAT_B8G8R8A8_UNORM
and src_format
being MESA_FORMAT_A8B8G8R8_UNORM
. When I print out parts of the src
and dst
buffers with:
for (int i = 0; i < 100; i+=4) {
fprintf(stderr, "%u %u %u %u X ", dst[i], dst[i+1], dst[i+2] ,dst[i+3]);
}
I see that they are mirrored (that is MESA_FORMAT_A8B8G8R8_UNORM
would suggest they will be ABGR, but in reality, it is RGBA; the same is true for the other buffer). But AFAICT, that is not the issue, and the fact that once the issue (below) is taken care of everything works only strengthens this assumption. Mutter also expects to get buffer in ARGB, which is again a mirror of MESA_FORMAT_B8G8R8A8_UNORM
, so everything matches.
(By mirrored I mean that the result from above loop print is 255(A) 24(R) 63(G) 167(B) X ...
.
But after the function call, dst
buffer is filled with bytes in RABG format, which is wrong no matter the expectations. So I investigated further and I believe that the bug is in _mesa_format_convert() in this part of the code:
} else if (src_array_format == RGBA8_UBYTE) {
assert(!_mesa_is_format_integer_color(dst_format));
if (dst_format == MESA_FORMAT_B8G8R8A8_UNORM) {
convert_ubyte_rgba_to_bgra(width, height, src, src_stride,
dst, dst_stride);
}
The reason being (my guess) that src_array_format() from
uint32_t
_mesa_format_to_array_format(mesa_format format)
{
const struct mesa_format_info *info = _mesa_get_format_info(format);
#if UTIL_ARCH_BIG_ENDIAN
if (info->ArrayFormat && info->Layout == MESA_FORMAT_LAYOUT_PACKED)
return _mesa_array_format_flip_channels(info->ArrayFormat);
else
#endif
return info->ArrayFormat;
}
does account for the fact that we are on big endian (and thus is RGBA8_UBYTE
, which is what I see when I dump the src
), but MESA_FORMAT_B8G8R8A8_UNORM
does not strictly reflect what is in the memory (and is mirrored in this case). This results in convert_ubyte_rgba_to_bgra()
being used and that is wrong. When I comment this block out, the issue is gone and I get ARGB, which is what mutter and gnome-screenshot expect.
This was found in Mesa 20.0.2, but looking at the code, there isn't much changed in these parts since, so it should still be an issue in the latest version (I will try to test that though).