xwayland: Confining the cursor (cursor grab) fails if the cursor is not over the window when the grab happens

added Xwayland label

When I move the mouse cursor to the second monitor and then start d1x-rebirth (with the controls set for using the mouse to "steer") then steering does not work until I move the mouse from the second monitor to the first one (where d1x is running fullscreen).

Xwayland does not control the pointer, so it cannot move the pointer back within the surface it it's placed outside, I guess that depends on the Wayland compositor.

Worse, I can also move the cursor back to the second monitor again which completely breaks the gameplay.

Oddly, I fail to reproduce this, but I reckon if the pointer locking/constrain is active, the Wayland compositor should ensure the pointer cannot leave the surface.

Could be interesting to capture the WAYLAND_DEBUG logs between Xwayland and the Wayland compositor when that occurs.

Oddly, I fail to reproduce this, but I reckon if the pointer locking/constrain is active, the Wayland compositor should ensure the pointer cannot leave the surface.

Right, the problem seems to be that the initial constrain fails and then we never retry, so moving the cursor over the Window results in mouse events being send while it is over the Window, but since the constrain fails it can leave the Window again. At least that is my theory.

Maybe we should set a flag on the xwl_window that a failed constrain is pending, catch mouse-enter events and then retry once the mouse is over the Window? This will mostly fix this (assuming my theory is valid), the initial constrain failing and the user requiring to move the mouse over the Window is not that much of a problem, it is the mouse being able to leave again afterwards which really breaks things.

Have you tried to reproduce without the xrandr patches, like with an unpatched build of Xwayland from upstream stable branch ?

I do not see any such issue with the standard unpatched stable build, so I wonder if that could be a problem introduced by the xrandr emulation patches (either in Xwayland or in mutter).

I tried to reproduce this with latest master + my recent set of randr emu fixes and I can no longer reproduce this either, so lets close this.

Nothing has changed in this regard in master AFAIK. This could be a race on mutter side.

FWIW, the constrain never fails on the client side:

https://gitlab.freedesktop.org/wayland/wayland-protocols/blob/master/unstable/pointer-constraints/pointer-constraints-unstable-v1.xml

    <request name="lock_pointer">
      <description summary="lock pointer to a position">
	The lock_pointer request lets the client request to disable movements of
	the virtual pointer (i.e. the cursor), effectively locking the pointer
	to a position. This request may not take effect immediately; in the
	future, when the compositor deems implementation-specific constraints
	are satisfied, the pointer lock will be activated and the compositor
	sends a locked event.

	The protocol provides no guarantee that the constraints are ever
	satisfied, and does not require the compositor to send an error if the
	constraints cannot ever be satisfied. It is thus possible to request a
	lock that will never activate.
	...
    </request>

Also, one cannot just issue another request:

https://gitlab.freedesktop.org/wayland/wayland-protocols/blob/master/unstable/pointer-constraints/pointer-constraints-unstable-v1.xml#L49

  <interface name="zwp_pointer_constraints_v1" version="1">
    <description summary="constrain the movement of a pointer">
      The global interface exposing pointer constraining functionality. It
      exposes two requests: lock_pointer for locking the pointer to its
      position, and confine_pointer for locking the pointer to a region.

      The lock_pointer and confine_pointer requests create the objects
      wp_locked_pointer and wp_confined_pointer respectively, and the client can
      use these objects to interact with the lock.

      For any surface, only one lock or confinement may be active across all
      wl_pointer objects of the same seat. If a lock or confinement is requested
      when another lock or confinement is active or requested on the same surface
      and with any of the wl_pointer objects of the same seat, an
      'already_constrained' error will be raised.
    </description>

Protocol errors are fatal to the Wayland clients, so that would kill Xwayland.

So there is not much Xwayland can do there, I'm afraid.

closed

Reopening the issue, this is a race somewhere that shows randomly.

FWIW, the input code hasn't changed in Xwayland since commit 6711b5c6 which was first included in xorg-server-1.20.6.

I wonder if commit 6711b5c6 might be related though, as it precisely touches the lock code.

reopened

mentioned in commit ofourdan/xserver@5387a6aa

@ofourdan, lets continue this here rather then by email so that others can chime in.

I have tried the Fedora xserver scratch build you provided by email, it seems to work a bit better, but I'm not sure it actually is better. It may just be that I understand the problem better now so it feels more predictable.

I have managed to make the problem 100% reproducible for me and there really are 2 different problems:

Problem 1: Sometimes the grab outright fails and I can move the cursor to my second monitor. This happens when I let d1x-rebirth run through its splash screen instead of pressing a key to skip the splash, to reproduce:

Take a dual monitor setup
Enable the rpmfusion nonfree repo (d1x-rebirth is based on the source release of the original descent game which comes with a no-commercial use license)
"sudo dnf install d1x" (note this comes with the original shareware datafiles)
Start d1x-rebirth create a player, enable mouse input in the controls
Start a xterm on the monitor at 0x0, put the cursor over the xterm
Start d1x-rebirth from the xterm, let it run through its splash screens
Once the select-player menu shows press enter a couple of times until you get the instructions for level 1. You can press ESC at the instructions to skip them
Move the mouse to the right (towards the second monitor) keep doing this until the ship stops turning, now you will see the mouse over the second monitor and if you click on a window on the second monitor the game leaves fullscreen. If you do this with the left mouse button pressed, the left mouse button gets stuck in the game.

It took me a while to realize that 2 conditions need to be true to reproduce this:

The mouse needs to be over another X11 window when the game starts
The game hides the cursor before it shows the initial splash and unhides it during the menu (I guess) when you skip the splash the timing is such that the cursor briefly shows over the d1x window during the menus even if you do not move the mouse, this is enough to avoid the bug. Another way of getting the same result is moving the cursor when the select-player or main menu is shown, this will also cause the cursor to be shown over the d1x window before the game tries to grab the mouse when the actual game starts.

So it seems that the problem of the grab not working is caused by the cursor last being drawn over (owned by?) another X11 window and it may be that the grab actually activates on the wrong window? Also see problem 2.

Note adding this as a comment now, I will describe problem 2 in a separate comment.

Start a xterm on the monitor at 0x0, put the cursor over the xterm ... Move the mouse to the left (towards the second monitor) keep doing this until the ship stops turning,

Can you post your output layout/config, that two sentences sound contradictory to me (0x0 is necessarily the leftmost output unless you have your output in the negative range which is hardly possible with gnome-shell/mutter)

The game hides the cursor before it shows the initial splash and unhides it during the menu (I guess) when you skip the splash the timing is such that the cursor briefly shows over the d1x window during the menus even if you do not move the mouse, this is enough to avoid the bug.

Cursor visibility is a determining factor for this to work, see #734 (closed)

Can you post your output layout/config, that two sentences sound contradictory to me

My bad, the "Move the mouse to the left" should be "Move the mouse to the right", will edit right away.

As for my layout it is 2 1920x1080 monitors side to side, SDL1 games like d1x always start on the monitor at 0x0 which is the left monitor, so to get to the second / other monitor I indeed need to move the mouse right (sorry).

OK, thanks, so this is very similar to my environment

Problem 2: When the cursor is on the second monitor and over an X11 window on that second monitor, then the grab does work but the cursor is confined to that X11 window on the second monitor.

Reproducing this is identical to the reproduction instructions for problem 1 except that you put the cursor over an X11 window (e.g. xterm, I noticed this with thunderbird) on the second monitor instead of over the xterm on the first monitor.

When this happens then the mouse works fine, you can keep moving it to the right for ever and the ship keeps turning circles. But as soon as you left click then the focus moves to the X11 window on the second monitor, d1x unfullscreens and things are broken(ish) from then on. The switch of focus also breaks the grab (which was on the wrong window), but I guess that might be d1x releasing it since it lost focus (the loose of focus is also why it unfullscreens).

Note I guess the root cause for these 2 problems is the same: if the cursor is (last seen) over an other X11 window the grab activates on that X11 window, which is the wrong X11 window.

Like with problem 1 I can work around problem 2 by moving the cursor during the player selection menu. In this case I need to move the cursor from monitor 2 to monitor 1 to workaround the issue.

I keep trying to reproduce following the steps you described to no avail... Neither problem 1 nor 2.

Do you have focus-follow-mouse or something peculiar set in gnome-shell to reproduce?

Ah! Looks like I've just reproduced with focus-follow-mouse set in gnome-shell :)

Weird, I do not have focus follow mouse set. I don't really have much of anything special. Pretty stock F32 with the 1.20.7-1.1test scratch-build you did.

This is actually a clean install which I did recently to test Fedora 32 on a X1 7th gen loaner.

mentioned in issue #1000 (closed)

~~And now the weirdest part... I cannot reproduce with !410 (merged) [1]~~

~~@jwrdegoede mind trying with !410 (merged)? If needed I can build a scratch build for F32/rawhide if that helps...~~

~~_{[1] weirdest, because I fail to see how that could be related...}~~

Nevermind...

OK so for problem 2. what happens is the following:

the cursor is placed on xterm
xterm hides the cursor while typing
the game is started from the xterm
xwl_seat_maybe_lock_on_hidden_cursor is triggered while on the xterm
the use clicks on button1
the event is sent to the window under the pointer which is still the xterm
focus is sent back to the xterm
xterm sets the cursor on pointer event
xwl_seat_cursor_visibility_changed is triggerd
xwl_seat_destroy_pointer_warp_emulator is called

And so ends tragically the lock...

Actually... For problem 2, attaching gdb to Xwayland and braking on xwl_seat_unconfine_pointer shows the pointer is unconfined because the game window is reparented, i.e. because it becomes decorated.

The next question would be why does mutter/gnome-shell reparent that window

The next question would be why does mutter/gnome-shell reparent that window

Well, because client asks for this, because the focus is set on the xterm:

FOCUS: Focusing 0x800022 due to button 1 press (display.c)
FOCUS: Setting input focus to window 0x800022, input: 1 focusable: 1
FOCUS: Setting input focus on 0x800022 since input = true
FOCUS: 0xc00010 is now the previous focus window due to being focused out or unmapped
VERBOSE: Grabbing unfocused window buttons for 0xc00010
FOCUS: * Focus --> 0x800022

~~Humm, that looks more and more like a mutter issue, it all behaves as if the confinement didn't work completely, as it it could be bypassed in some cases.~~

Rather, a race between the client (the game) which shows/hides the cursor and Xwayland which uses either pointer locking or confinement depending on the cursor visibility.

That code from commit a6e85e63 looks wrong:

https://gitlab.freedesktop.org/xorg/xserver/blob/master/hw/xwayland/xwayland-input.c#L2614

2597 static void
2598 xwl_pointer_warp_emulator_set_fake_pos(struct xwl_pointer_warp_emulator *warp_emulator,
2599                                        int x,
2600                                        int y)
2601 {
...
2613     window = warp_emulator->xwl_seat->focus_window->window;
2614     if (x >= window->drawable.x ||
2615         y >= window->drawable.y ||
2616         x < (window->drawable.x + window->drawable.width) ||
2617         y < (window->drawable.y + window->drawable.height)) {
2618         sx = x - window->drawable.x;
2619         sy = y - window->drawable.y;
2620         zwp_locked_pointer_v1_set_cursor_position_hint(locked_pointer,
2621                                                        wl_fixed_from_int(sx),
2622                                                        wl_fixed_from_int(sy));
2623         wl_surface_commit(warp_emulator->xwl_seat->focus_window->surface);
2624     }
...

Granted, it's unlikely the problem, but that still looks suspicious.

Actually, it's not, its just a hint for the compositor as to where to place the cursor upon unlock. Nevermind.

Humm, actually, I think I might have found the issue...

I was wrong from the start for two reasons:

I had !395 applied on my tree and that breaks in creative ways.
More importantly, I was expecting problem 2 to work.

Basically, I've spent countless hour trying so solve problem 2 while actually this is... somehow expected.

The pointer gets locked on the xterm, outside of the game window. As soon as a click is emitted, that event is routed to the xterm, which gets the focus and the game leaves fullscreen. All that is normal and happens as well when the terminal is a Wayland native client (e.g. gnome-terminal).

The actual problem is that the pointer remains invisible and that's because the pointer is locked on the xterm window.

So instead of trying to solve problem 2, we should make it fail more clearly, basically, behave like with Wayland native client.

What I mean by that is, if the cursor confinement window is different from the focus window, we should not try to lock the pointer when hiding the cursor. That allows the user to move the pointer back to the game window and operate the game smoothly.

Of course, considering that the game sets an empty cursor, the cursor remains invisible while within the xterm, but the pointer can be moved outside the window and reach the game window on the other monitor. Yet, clicking outside the game window will move focus and the game will leave fullscreen, this is normal and expected.

And I suspect problem 1 is also related.

mentioned in commit ofourdan/xserver@723a48a3

xwayland: Confining the cursor (cursor grab) fails if the cursor is not over the window when the grab happens

Child items ...

Activity

Admin message

Admin message

xwayland: Confining the cursor (cursor grab) fails if the cursor is not over the window when the grab happens

Activity