Reference count overflow in shm leads to use-after-free
The shared memory code uses an int for the reference count. On 64-bit systems, a malicous client can create so many references that the int overflows. This is undefined behavior, but it will most likely cause the int to overflow, causing an exploitable use-after-free. A successful exploit will result in the execution of arbitrary code in the context of the Wayland compositor.
To fix this bug, uintptr_t (or intptr_t) should be used for all reference counts. There are only UINT_MAX possible distinct pointers, so a uintptr_t reference count can never overflow, and an intptr_t reference count cannot overflow if the referencing and referenced objects are at least 2 bytes. This assumes that INTPTR_MAX == (UINTPTR_MAX >> 1) && UINTPTR_MAX == ((uintptr_t)INTPTR_MAX << 1) + 1, which can be checked with a C11 _Static_assert.