[Backport to Xwayland 23.1] Backport the pointer fixes

Merged Olivier Fourdan requested to merge ofourdan/xserver:backport-xwayland-23.1 into xwayland-23.1

This backports !1008 (merged) to xwayland-23.1

Merge request reports