xwayland/shm: Avoid integer overflow on large pixmaps

Xwayland's xwl_shm_create_pixmap() computes the size of the shared
memory pool to create using a size_t, yet the Wayland protocol uses an
integer for that size.

If the pool size becomes larger than INT32_MAX, we end up asking Wayland
to create a shared memory pool of negative size which in turn will raise
a protocol error which terminates the Wayland connection, and therefore
Xwayland.

Avoid that issue early by return a NULL pixmap in that case, which will
trigger a BadAlloc error, but leave Xwayland alive.

Signed-off-by: Olivier Fourdan <ofourdan@redhat.com>
Reviewed-by: Jonas Ådahl <jadahl@gmail.com>
5 jobs for xwayland-shm-overflow in 3 minutes and 52 seconds (queued for 17 seconds)
Status Name Job ID Coverage
  Docker Image
passed debian-buster #14361106

00:00:09

 
  Build And Test
passed autotools #14361107

00:03:42

passed meson #14361108

00:02:09

passed meson-noglamor #14361109

00:01:50

passed mingw-cross-build #14361110

00:00:44