• Leandro Lupori's avatar
    xserver: fix RGB mask handling · 7e142cb2
    Leandro Lupori authored and Povilas Kanapickas's avatar Povilas Kanapickas committed
    On FreeBSD 13.0-CURRENT for PowerPC64 big-endian (BE), X was
    crashing in some cases. For instance, when twm was started
    and the background was clicked to open its menu, X crashed
    with a segmentation fault, trying to dereference a null pointer
    at CreatePicture().
    There were 2 issues with xorg-server handling of RGB masks that
    caused the pointer above to be null and thus the crash:
    - wrong use of ffs() to get the RGB offsets from the masks
    - overflow when shifting a 16-bit integer
    This change fixes both issues. They happen when the system is BE
    but has a video adapter using a little-endian (LE) ARGB32
    framebuffer. In order to display the correct colors, this setup
    requires a BE RGBA32 color format to be used by X, by setting
    the RGB masks appropriately, that didn't work properly because of
    the issues above.
xf86Helper.c 49.1 KB