xwayland-glx: Fix GLX visual mask setup

a2rgb10 configs would end up with channel masks corresponding to
argb8888. This would confuse the GLX core code into matching an a2rgb10
config to the root window visual, and that would make things look wrong
and bad.

Fix this by handling more cases. We're still not fully general here, and
this could still be wrong on big-endian. The XXX comment about doing
something less ugly still applies, ideally we would get this information
out of EGL instead of making lucky guesses. Still, better than it was.

Fixes: xorg/xserver#824
3 jobs for eglx-fix-rgb10 in 5 minutes and 35 seconds (queued for 1 second)