[DCN301/Vangogh] Toggling overlay plane (i.e. MangoHud performance overlay in Steam) causes artifacts to briefly appear on the display within gamescope-session (kernel 6.5 and later)

Brief summary of the problem:

When invoking or clearing overlay planes on the Steam Deck LCD and Steam Deck OLED within gamescope-session (Game Mode) on kernels past 6.5, there is a brief period where artifacting is introduced to the primary plane - but only when there is an ongoing cycle of

[ 1188.065190] amdgpu 0000:04:00.0: [drm:drm_atomic_get_crtc_state] Added [CRTC:92:crtc-0] 00000000767a3699 state to 00000000c358238c
[ 1188.065198] amdgpu 0000:04:00.0: [drm:drm_atomic_set_fb_for_plane] Set [FB:143] for [PLANE:71:plane-3] state 00000000024c6fd4
[ 1188.065210] amdgpu 0000:04:00.0: [drm:drm_atomic_get_plane_state] Added [PLANE:77:plane-4] 00000000d72e7b30 state to 00000000c358238c
[ 1188.065215] amdgpu 0000:04:00.0: [drm:drm_atomic_set_fb_for_plane] Set [NOFB] for [PLANE:77:plane-4] state 00000000d72e7b30

such as when content is being displayed. Gamescope also MUST be using direct scan-out (not actively compositing) to make this happen. You can verify this with the Composite Debug option in the Steam developer settings - no squares should appear when toggled. If they do, this bug will not be reproable.

As far as I can tell, this is a kernel regression that took place sometime after 6.1 and before 6.5, since I am unable to repro the issue on 6.1.52-valve19. Valve's 6.5.0-valve kernel is affected by this bug, so is my standard Fedora kernel, and it happens all the way up to agd5f/linux@aae0be56 on amd-staging-drm-next. Bisecting for it is a bit tricky since the Steam Deck will default to always compositing if you have an unpatched kernel under 6.5.

I'm including videos down below, along with drm debugging logs that will show the series of events that takes place to make this artifacting occur. Please note - this is separate from the explicit sync artifacting I reported previously.

Hardware description:

CPU: AMD Custom 0932 (8) @ 2.80 GHz
GPU: Advanced Micro Devices, Inc. [AMD/ATI] Sephiroth [AMD Custom GPU 0405] [1002:1435] (rev ae)
System Memory: 16GB
Display(s): internal display
Type of Display Connection: eDP

System information:

Distro name and Version: Fedora 40, SteamOS 3.6/3.7
Kernel version: 6.5.0 -> 6.9.5
Custom kernel: 6.9.5.fsync kernel and amd-staging-drm-next for Fedora 40, 6.5.0-valve and 6.8.0-valve-preview kernels for SteamOS
AMD official driver version: Mesa 24.1.1

How to reproduce the issue:

The same steps apply for both Valve's 6.5.0-valve11 kernel as well as upstream amd-staging-drm-next

Enter gamescope-session
Click on a game entry from the Home screen
Leave the game entry
If you're watching your drm debugging logs, you should be seeing an active cycling of planes.
While this cycling is ongoing, toggle the steam performance overlay on and off repeatedly from the right side menu

You should see artifacts at this point.

Attached files:

Logs/video files

Starting with logs and a video from 6.1.52-valve19:

Disabling the overlay:

  625.145145] [drm:dm_update_plane_state [amdgpu]] Disabling DRM plane: 90 on DRM crtc 100
[  625.145425] amdgpu 0000:04:00.0: [drm:drm_atomic_get_private_obj_state] Added new private object 000000004db1c4e7 state 000000002df8647f to 00000000ff8a9ab1
[  625.145429] [drm:dm_update_plane_state [amdgpu]] Disabling DRM plane: 84 on DRM crtc 100
[  625.145670] [drm:dm_update_plane_state [amdgpu]] Disabling DRM plane: 72 on DRM crtc 100
[  625.145932] [drm:dm_update_plane_state [amdgpu]] Enabling DRM plane: 90 on DRM crtc 100
[  625.146177] [drm:dm_crtc_helper_atomic_check [amdgpu]] Can't enable a CRTC without enabling the primary plane
[  625.146464] amdgpu 0000:04:00.0: [drm:drm_atomic_helper_check_planes] [CRTC:100:crtc-0] atomic driver check failed
[  625.146469] [drm:amdgpu_dm_atomic_check [amdgpu]] drm_atomic_helper_check_planes() failed
[  625.146743] [drm:amdgpu_dm_atomic_check [amdgpu]] Atomic check failed with err: -22

Enabling the overlay:

[  700.101125] [drm:dm_update_plane_state [amdgpu]] Disabling DRM plane: 90 on DRM crtc 100
[  700.101401] amdgpu 0000:04:00.0: [drm:drm_atomic_get_private_obj_state] Added new private object 000000004db1c4e7 state 000000003dd68ca8 to 0000000030834f76
[  700.101405] [drm:dm_update_plane_state [amdgpu]] Disabling DRM plane: 72 on DRM crtc 100
[  700.101668] [drm:dm_update_plane_state [amdgpu]] Enabling DRM plane: 90 on DRM crtc 100
[  700.102029] [drm:dm_update_plane_state [amdgpu]] Enabling DRM plane: 84 on DRM crtc 100
[  700.102395] [drm:dm_update_plane_state [amdgpu]] Enabling DRM plane: 72 on DRM crtc 100
[  700.102654] [drm:amdgpu_dm_atomic_check [amdgpu]] MPO enablement requested on crtc:[00000000cc319b52]
[  700.102952] amdgpu 0000:04:00.0: [drm:drm_atomic_nonblocking_commit] committing 0000000030834f76 nonblocking
[  700.102959] [drm:dm_plane_helper_prepare_fb [amdgpu]] No FB bound
[  700.103202] [drm:dm_plane_helper_prepare_fb [amdgpu]] No FB bound
[  700.103442] [drm:dm_plane_helper_prepare_fb [amdgpu]] No FB bound
[  700.103688] [drm:dm_plane_helper_prepare_fb [amdgpu]] No FB bound
[  700.103933] [drm:dm_plane_helper_prepare_fb [amdgpu]] No FB bound
[  700.104172] [drm:dm_plane_helper_prepare_fb [amdgpu]] No FB bound
[  700.104411] [drm:dm_plane_helper_prepare_fb [amdgpu]] No FB bound
[  700.104652] [drm:dm_plane_helper_prepare_fb [amdgpu]] No FB bound
[  700.105267] [drm:dcn301_smu_set_min_deep_sleep_dcfclk [amdgpu]] dcn301_smu_set_min_deep_sleep_dcfclk(17832)
[  700.105754] [drm:dcn301_smu_set_dppclk [amdgpu]] dcn301_smu_set_dppclk(102857)
[  700.106507] [drm:dcn20_program_front_end_for_ctx [amdgpu]] Reset mpcc for pipe 1
[  700.114021] [drm:mpc2_assert_idle_mpcc [amdgpu]] REG_WAIT taking a while: 3ms in mpc2_assert_idle_mpcc line:478
[  700.114419] [drm:dcn10_plane_atomic_power_down [amdgpu]] Power gated front end 1
[  700.114793] [drm:dcn20_post_unlock_program_front_end [amdgpu]] Power down front end 1

Video clip (no artifacting present at any point): https://www.youtube.com/watch?v=xaAIxSn_8co

The videos were too large to attach to this issue without heavy compression which makes it more difficult to see the artifacting, so apologies for that.

Now here are my logs from 6.5.0-valve11: