wayland-shm: Keep shm pool data mappings valid.

Erik De Rijcke requested to merge zubzub/wayland:shm-resize-fixes into main

This PR introduces a new way of dealing with shm pools that are resized while the compositor has taken an external ref and stored an external data pointer into the pool.

The problem:

The previous solution would defer the resizing of the pool when an external ref was present because a pool resize would invalidate any previous external data pointers into the pool. What happens is that a client is unaware of this defered resize and will happily try to create a new shm buffer that is beyond the bounds of the to-be-resized shm pool which causes the client to be disconnected because of a protocol violation. See also https://gitlab.gnome.org/GNOME/gtk/-/issues/4460#note_1314159

Other relevant past discussion: https://lists.freedesktop.org/archives/wayland-devel/2016-February/026905.html

The solution:

Instead of working around this problem (which imho is actually a wayland protocol violation as client buffers are not to be fiddled with once committed but this is how it is working for a long time now...), the solution lies in keeping the previous mapping valid after a pool resize has taken place. This can done by specifying an 'old_size' of '0' in the 'mremap' call. As per mremap man-page:

If the value of old_size is zero, and old_address refers to a shareable mapping (see mmap(2) MAP_SHARED), then mremap() will create a new mapping of the same pages.

This keeps the previous mapping around and valid. This PR keeps track of these previous mappings and unmaps them automatically once the external refcount reaches zero.

Edited by Erik De Rijcke

Merge request reports