Skip to content

Hard system lockup on 6.7-rc1 or newer kernel with 6600 XT when using capturing screen with vaapi encoding

Brief summary of the problem:

System hard lockup when recording the screen and encoding it with vaapi, using applications such as Vesktop.

Affected kernels: Kernel 6.7-rc1 or newer, up to the latest kernel (6.9.1)

Not affected kernels: Any kernel from the current LTS 6.6.x series.

Hardware description:

  • CPU: i5-13600k
  • GPU: RX 6600 XT
  • Type of Display Connection: DP for 2 monitors

System information:

  • Distro name and Version: openSUSE Tumbleweed
  • Mesa version: Tried with both Mesa master and the latest tagged release at multiple points, not related to Mesa.

How to reproduce the issue:

I'm not a 100% sure what the necessary parts to reproduce the issue are, but recording your screen with vaapi encoding will cause a complete system lock up within 5-60 minutes.

Log files (for system lockups / game freezes / crashes)

The system locks up before there are any logs printed, there's no errors in dmesg or anything else. It even corrupts files being written at the time, such as bash history.

Attempted bisect

git bisect start
# status: waiting for both good and bad commits
# bad: [0dd3ee31125508cd67f7e7172247f05b7fd1753a] Linux 6.7
git bisect bad 0dd3ee31125508cd67f7e7172247f05b7fd1753a
# status: waiting for good commit(s), bad commit known
# good: [ffc253263a1375a65fa6c9f62a893e9767fbebfa] Linux 6.6
git bisect good ffc253263a1375a65fa6c9f62a893e9767fbebfa
# bad: [deefd5024f0772cf56052ace9a8c347dc70bcaf3] Merge tag 'vfio-v6.7-rc1' of https://github.com/awilliam/linux-vfio
git bisect bad deefd5024f0772cf56052ace9a8c347dc70bcaf3
# good: [5a6a09e97199d6600d31383055f9d43fbbcbe86f] Merge tag 'cgroup-for-6.7' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup
git bisect good 5a6a09e97199d6600d31383055f9d43fbbcbe86f
# good: [59fff63cc2b75dcfe08f9eeb4b2187d73e53843d] Merge tag 'platform-drivers-x86-v6.7-1' of git://git.kernel.org/pub/scm/linux/kernel/git/pdx86/platform-drivers-x86
git bisect good 59fff63cc2b75dcfe08f9eeb4b2187d73e53843d
# bad: [27442758e9b4e083bef3f164a1739475c01f3202] Merge tag 'amd-drm-next-6.7-2023-10-13' of https://gitlab.freedesktop.org/agd5f/linux into drm-next
git bisect bad 27442758e9b4e083bef3f164a1739475c01f3202

I'm quite certain 27442758e9b4e083bef3f164a1739475c01f3202 is the first bad commit, but I can't actually bisect the individual commits within the tag because almost 90% of the commits I build don't display anything on boot, even though my system is running and I can ssh into it.