xserver: fix RGB mask handling
On FreeBSD 13.0-CURRENT for PowerPC64 big-endian (BE), X was crashing in some cases. For instance, when twm was started and the background was clicked to open its menu, X crashed with a segmentation fault, trying to dereference a null pointer at CreatePicture().
There were 2 issues with xorg-server handling of RGB masks that caused the pointer above to be null and thus the crash:
- wrong use of ffs() to get the RGB offsets from the masks
- overflow when shifting a 16-bit integer
This change fixes both issues. They happen when the system is BE but has a video adapter using a little-endian (LE) ARGB32 framebuffer. In order to display the correct colors, this setup requires a BE RGBA32 color format to be used by X, by setting the RGB masks appropriately, that didn't work properly because of the issues above.