Playback issues with libva on Kabylake GT2/modesetting
A new issue appeared in Xorg 1.20.1, which didn't exist in 1.20.0.
On a lenovo yoga 1st or 3rd gen (skylake or kabylake), both using the modesetting driver, playback using libva (using mpv, ffmpeg, ...) works, but causes the display to display a freezed image if I switch desktop and/or move the playback window to another desktop while the video is playing.
I verified the issue with kernels 4.17 to 4.19-rc4.
Once the screen stops updating, there's not much one can do except kill xorg.
I discovered I can "unfix" the screen by triggering a display rotation by 90 degrees, which shows the image instantly, and rotating it back. 1.20.0 introduced some fixes for display rotation: https://bugs.freedesktop.org/show_bug.cgi?id=106715 which I suspect are somewhat related. I reported another bug with this intel/kms combination about a similar issue meta#8 (moved) which is also "fixed" by performing a display rotation.
Doing a 180 degree rotation is not sufficient. A 90 degree rotation is required.
There's no error or informational message being written to dmesg or Xorg.log. The system is otherwise responsive, and xorg also appears to be working (since I can rotate the display).
The behavior is the same if one or two screens are attached (built-in display + external or built-in only).
I reverted to xserver-xorg-core 1.19.99 (from debian snapshots) due to a number of outstanding bugs in the 1.20.* series, keeping everything else the same (libdrm, kernel, etc), and the issue is gone.