X crashes when video driver uses a big-endian RGBA32 color format
On a machine running FreeBSD 13.0-CURRENT for PowerPC64 big-endian (BE), X (xorg-server-1.20.9) is crashing when, for instance, twm is started and I click the background to open its menu:
(EE) Backtrace:
(EE) 0: /usr/local/bin/Xorg (?+0x0) [0x10284234]
(EE) unw_get_proc_name failed: no unwind info found [-10]
(EE) 1: /lib/libthr.so.3 (?+0x0) [0x810a0bee4]
(EE) unw_get_proc_name failed: no unwind info found [-10]
(EE) 2: /lib/libthr.so.3 (?+0x0) [0x810a0b080]
(EE) unw_get_proc_info failed: no unwind info found [-10]
(EE)
(EE) Segmentation fault at address 0x4
(EE)
Fatal server error:
(EE) Caught signal 11 (Segmentation fault). Server aborting
This is the stacktrace captured by GDB, in a build with debug symbols:
Thread 1 "MainThread" received signal SIGSEGV, Segmentation fault.
0x000000001025be04 in CreatePicture (pid=0, pDrawable=0x81e5d12c0, pFormat=0x0, vmask=0, vlist=0x0,
client=0x811454000, error=0xfffffbfffe4e4) at picture.c:763
763 pPicture->format = pFormat->format | (pDrawable->bitsPerPixel << 24);
(gdb) bt
#0 0x000000001025be04 in CreatePicture (pid=0, pDrawable=0x81e5d12c0, pFormat=0x0, vmask=0, vlist=0x0,
client=0x811454000, error=0xfffffbfffe4e4) at picture.c:763
#1 0x00000000101d0ebc in compWindowUpdateAutomatic (pWin=0x81150d700) at compwindow.c:669
#2 0x00000000101d0700 in compPaintWindowToParent (pWin=0x81150d700) at compwindow.c:729
#3 0x00000000101d0638 in compPaintChildrenToWindow (pWin=0x811541900) at compwindow.c:744
#4 0x00000000101cc1c8 in compScreenUpdate (pClient=0x811454000, closure=0x811532600) at compalloc.c:57
#5 0x00000000100d01e0 in ProcessWorkQueue () at dixutils.c:536
#6 0x0000000010367638 in WaitForSomething (are_ready=0) at WaitFor.c:192
#7 0x00000000100b83e8 in Dispatch () at dispatch.c:421
#8 0x00000000100ce46c in dix_main (argc=4, argv=0xfffffbfffeaf8, envp=0xfffffbfffeb20) at main.c:276
#9 0x00000000100a3880 in main (argc=4, argv=0xfffffbfffeaf8, envp=0xfffffbfffeb20) at stubmain.c:34
This is a Talos II machine from Raptor (https://raptorcs.com/content/TL2DS1/intro.html), that has a POWER9 CPU. It has a built in video adapter uses a little-endian (LE) ARGB32 framebuffer. This setup requires a BE RGBA32 color format to be used by X, to display the correct colors.
This setup is done by FreeBSD's xf86-video-scfb driver, that calls xf86SetWeight() with a mask value corresponding to a BE RGBA32 color format (although this change is not yet upstreamed and currently it uses only the default, all zeroes, mask).
After some debugging, I've found the root cause of the segmentation fault above: it happens because there are 2 issues with xorg-server handling of RGB masks. The first is in xf86Helper.c, that is using ffs() to get the RGB offsets from the masks, but without subtracting 1 from the result, that is needed because ffs() returns 1 for the least significant bit. The second is in render/picture.c, where a 16-bit integer may be shifted by more than 16 bits, resulting in a wrong value.
The attached patches fix both issues. I've tested them in a BE FreeBSD install as well as in a LE install and they seem to work fine.