xserver: fix RGB mask handling

On FreeBSD 13.0-CURRENT for PowerPC64 big-endian (BE), X was
crashing in some cases. For instance, when twm was started
and the background was clicked to open its menu, X crashed
with a segmentation fault, trying to dereference a null pointer
at CreatePicture().

There were 2 issues with xorg-server handling of RGB masks that
caused the pointer above to be null and thus the crash:
- wrong use of ffs() to get the RGB offsets from the masks
- overflow when shifting a 16-bit integer

This change fixes both issues. They happen when the system is BE
but has a video adapter using a little-endian (LE) ARGB32
framebuffer. In order to display the correct colors, this setup
requires a BE RGBA32 color format to be used by X, by setting
the RGB masks appropriately, that didn't work properly because of
the issues above.
4 jobs for fix-rgb-mask-handling in 6 minutes and 17 seconds (queued for 2 seconds)
Status Name Job ID Coverage
  Docker Image
passed debian-buster #8945973

00:00:15

 
  Build And Test
passed autotools #8945974

00:06:01

passed meson #8945975

00:02:43

passed meson-noglamor #8945976

00:02:27