Vulkan shows stuttering issues on Vega 10 w/ gnome-shell on Wayland

added Xwayland label

It's a race between the Wayland server reading from the buffer and Xwayland drawing to it. This can result in tearing if those two operations can actually run concurrently, which is the case with amdgpu, but not yet with most other drivers AFAIK.

Either way, it can result in the Wayland server already displaying a client frame too early sometimes, hence the stuttering.

Right, I'm sorry, it's still unclear to me, still trying to understand the issue...

Xwayland should not access the buffer after it's been committed and before it's released by the compositor:

Surface contents are double-buffered state, see wl_surface.commit.

The initial surface contents are void; there is no content. wl_surface.attach assigns the given wl_buffer as the pending wl_buffer. wl_surface.commit makes the pending wl_buffer the new surface contents, and the size of the surface becomes the size calculated from the wl_buffer, as described above. After commit, there is no pending buffer until the next attach.

Committing a pending wl_buffer allows the compositor to read the pixels in the wl_buffer. The compositor may access the pixels at any time after the wl_surface.commit request. When the compositor will not access the pixels anymore, it will send the wl_buffer.release event. Only after receiving wl_buffer.release, the client may reuse the wl_buffer. A wl_buffer that has been attached and then replaced by another attach instead of committed will not receive a release event, and is not used by the compositor.

And:

Committing a pending wl_buffer allows the compositor to read the pixels in the wl_buffer. The compositor may access the pixels at any time after the wl_surface.commit request. When the compositor will not access the pixels anymore, it will send the wl_buffer.release event. Only after receiving wl_buffer.release, the client may reuse the wl_buffer. A wl_buffer that has been attached and then replaced by another attach instead of committed will not receive a release event, and is not used by the compositor.

Xwayland will attach the wl_surface on damage in xwl_window_post_damage() , set up the frame callback that will make sure we don't attach a new buffer if pending.

So we should not end up with the Wayland compositor reading from the buffer and Xwayland drawing to it at the same time.

The problem is that Xwayland only uses a single buffer per top-level X window (struct xwl_window).¹ So while you're right that it waits for the frame (not buffer release) callback, it keeps re-attaching the same buffer (so the buffer release callback would never happen anyway), and further drawing operations can hit the same buffer at any time.

The solution would be to alternate between at least two separate buffers per xwl_window, using some kind of (probably damage based) scheme to keep contents consistent between them and X11 drawing operations, while only doing drawing to buffers which are not currently in use by the Wayland server (i.e. which have either not been attached yet, or have had a buffer release callback).