Skip to content

xwayland/shm: Avoid integer overflow on large pixmaps

Olivier Fourdan requested to merge ofourdan/xserver:xwayland-shm-overflow into master

Xwayland's xwl_shm_create_pixmap() computes the size of the shared memory pool to create using a size_t, yet the Wayland protocol uses an integer for that size.

If the pool size becomes larger than INT32_MAX, we end up asking Wayland to create a shared memory pool of negative size which in turn will raise a protocol error which terminates the Wayland connection, and therefore Xwayland.

Avoid that issue early by return a NULL pixmap in that case, which will trigger a BadAlloc error, but leave Xwayland alive.

Signed-off-by: Olivier Fourdan ofourdan@redhat.com

Merge request reports