Picking DoubleScan resolutions result in kernel-level system hang

When I use xrandr or KDE to set a CRT monitor to a mode with DoubleScan enabled, this causes the mode selection to fail, and sometimes lock up the entire system. I have to REISUB to get out.

1. Power on a VGA CRT and connect it to a DP-to-VGA adapter (in any order).
2. Plug the DP-to-VGA adapter (connected to a powered-on CRT) into a DP port on your GPU. (My CRT does not emit EDID data through a plugged-in VGA cable when powered off, which is not good behavior but you can't exactly ask Gateway to release a fixed CRT these days.)
3. xrandr --output DP-2 --mode 0x9c

• If you're using KDE, in "Display Configuration", set the CRT to 800x600 at 60 Hz and Apply.

This selects mode:

 800x600 (0x9c) 81.000MHz +HSync +VSync DoubleScan  
       h: width   800 start  832 end  928 total 1080 skew    0 clock  75.00KHz  
       v: height  600 start  600 end  602 total  625           clock  60.00Hz  

Resulting in error:

Apr 08 04:23:16 ryzen kernel: ------------[ cut here ]------------
Apr 08 04:23:16 ryzen kernel: WARNING: CPU: 11 PID: 857 at drivers/gpu/drm/amd/amdgpu/../display/dc/core/dc.c:1225 dc_commit_state_no_check+0x15be/0x18d0 [amdgpu]
Apr 08 04:23:16 ryzen kernel: Modules linked in: snd_seq_dummy snd_hrtimer snd_seq rfcomm cmac algif_hash algif_skcipher af_alg bnep btusb btrtl btbcm btintel btmtk bluetooth ecdh_generic crc16 joydev mousedev xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tc>
Apr 08 04:23:16 ryzen kernel:  snd_timer rapl drm_display_helper sp5100_tco psmouse snd cec pcspkr ccp r8168(OE) soundcore i2c_piix4 zenpower(OE) wmi gpio_amdpt gpio_generic acpi_cpufreq mac_hid uinput dm_multipath it87(OE) hwmon_vid sg dm_mod crypto_user fuse loop bpf_preload ip>
Apr 08 04:23:16 ryzen kernel: CPU: 11 PID: 857 Comm: Xorg Tainted: G S         OE      6.2.9-zen1-1-zen #1 4d1dd606b844e45b5537702995754c641d4e30f1
Apr 08 04:23:16 ryzen kernel: Hardware name: Gigabyte Technology Co., Ltd. B550M DS3H/B550M DS3H, BIOS F15c 05/11/2022
Apr 08 04:23:16 ryzen kernel: RIP: 0010:dc_commit_state_no_check+0x15be/0x18d0 [amdgpu]
Apr 08 04:23:16 ryzen kernel: Code: ff 89 c6 e9 ce f5 ff ff 48 89 ef e8 4c 99 01 00 48 89 ef e8 34 d1 76 f0 e9 8d f6 ff ff 80 b8 80 03 00 00 00 0f 84 aa f4 ff ff <0f> 0b e9 a3 f4 ff ff 31 c0 e9 a0 f5 ff ff be 03 00 00 00 e8 aa 6d
Apr 08 04:23:16 ryzen kernel: RSP: 0018:ffffba68c177f758 EFLAGS: 00010202
Apr 08 04:23:16 ryzen kernel: RAX: ffff9d5250913c00 RBX: ffff9d5259440000 RCX: 000000000000000b
Apr 08 04:23:16 ryzen kernel: RDX: 0000000000001642 RSI: 0000000000000e6e RDI: 00000020b6597619
Apr 08 04:23:16 ryzen kernel: RBP: ffff9d5259440aa0 R08: ffffba68c177f724 R09: ffff9d5259440000
Apr 08 04:23:16 ryzen kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Apr 08 04:23:16 ryzen kernel: R13: ffff9d5259443638 R14: ffff9d51e56e0000 R15: 0000000000000003
Apr 08 04:23:16 ryzen kernel: FS:  00007f5bee5c5440(0000) GS:ffff9d54ef0c0000(0000) knlGS:0000000000000000
Apr 08 04:23:16 ryzen kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 08 04:23:16 ryzen kernel: CR2: 000055ccc61fffe0 CR3: 000000013ac88000 CR4: 0000000000750ee0
Apr 08 04:23:16 ryzen kernel: PKRU: 55555554
Apr 08 04:23:16 ryzen kernel: Call Trace:
Apr 08 04:23:16 ryzen kernel:  <TASK>
Apr 08 04:23:16 ryzen kernel:  dc_commit_state+0x186/0x1c0 [amdgpu a1822aa2219dc76b20c2f43ada3749139270af48]
Apr 08 04:23:16 ryzen kernel:  amdgpu_dm_atomic_commit_tail+0x3b0/0x3100 [amdgpu a1822aa2219dc76b20c2f43ada3749139270af48]
Apr 08 04:23:16 ryzen kernel:  ? free_unref_page+0x334/0x700
Apr 08 04:23:16 ryzen kernel:  ? bw_calcs+0x803/0x1f80 [amdgpu a1822aa2219dc76b20c2f43ada3749139270af48]
Apr 08 04:23:16 ryzen kernel:  ? dce112_validate_bandwidth+0x76/0x1c0 [amdgpu a1822aa2219dc76b20c2f43ada3749139270af48]
Apr 08 04:23:16 ryzen kernel:  ? dc_validate_global_state+0x3db/0x580 [amdgpu a1822aa2219dc76b20c2f43ada3749139270af48]
Apr 08 04:23:16 ryzen kernel:  ? dma_resv_get_fences+0xa3/0x2c0
Apr 08 04:23:16 ryzen kernel:  ? dma_resv_get_singleton+0x46/0x140
Apr 08 04:23:16 ryzen kernel:  ? wait_for_completion_timeout+0x13e/0x170
Apr 08 04:23:16 ryzen kernel:  ? wait_for_completion_interruptible+0x139/0x1e0
Apr 08 04:23:16 ryzen kernel:  commit_tail+0x94/0x130
Apr 08 04:23:16 ryzen kernel:  drm_atomic_helper_commit+0x116/0x140
Apr 08 04:23:16 ryzen kernel:  drm_atomic_commit+0x9a/0xd0
Apr 08 04:23:16 ryzen kernel:  ? __pfx___drm_printfn_info+0x10/0x10
Apr 08 04:23:16 ryzen kernel:  drm_atomic_helper_set_config+0x74/0xb0
Apr 08 04:23:16 ryzen kernel:  drm_mode_setcrtc+0x453/0x8c0
Apr 08 04:23:16 ryzen kernel:  ? __pfx_drm_mode_setcrtc+0x10/0x10
Apr 08 04:23:16 ryzen kernel:  drm_ioctl_kernel+0xcd/0x170
Apr 08 04:23:16 ryzen kernel:  drm_ioctl+0x233/0x410
Apr 08 04:23:16 ryzen kernel:  ? __pfx_drm_mode_setcrtc+0x10/0x10
Apr 08 04:23:16 ryzen kernel:  amdgpu_drm_ioctl+0x4e/0x90 [amdgpu a1822aa2219dc76b20c2f43ada3749139270af48]
Apr 08 04:23:16 ryzen kernel:  __x64_sys_ioctl+0x94/0xd0
Apr 08 04:23:16 ryzen kernel:  do_syscall_64+0x5f/0x90
Apr 08 04:23:16 ryzen kernel:  ? exc_page_fault+0x74/0x170
Apr 08 04:23:16 ryzen kernel:  entry_SYSCALL_64_after_hwframe+0x72/0xdc
Apr 08 04:23:16 ryzen kernel: RIP: 0033:0x7f5beef9653f
Apr 08 04:23:16 ryzen kernel: Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 18 48 8b 44 24 18 64 48 2b 04 25 28 00 00
Apr 08 04:23:16 ryzen kernel: RSP: 002b:00007fffde64ad00 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Apr 08 04:23:16 ryzen kernel: RAX: ffffffffffffffda RBX: 000055b14cc48b40 RCX: 00007f5beef9653f
Apr 08 04:23:16 ryzen kernel: RDX: 00007fffde64ad90 RSI: 00000000c06864a2 RDI: 0000000000000012
Apr 08 04:23:16 ryzen kernel: RBP: 00007fffde64ad90 R08: 0000000000000000 R09: 000055b14cc604b0
Apr 08 04:23:16 ryzen kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00000000c06864a2
Apr 08 04:23:16 ryzen kernel: R13: 0000000000000012 R14: 000055b14ca689c8 R15: 00007fffde64ae40
Apr 08 04:23:16 ryzen kernel:  </TASK>
Apr 08 04:23:16 ryzen kernel: ---[ end trace 0000000000000000 ]---

...

Apr 08 04:23:26 ryzen kernel: amdgpu 0000:05:00.0: [drm] *ERROR* [CRTC:55:crtc-1] flip_done timed out

...

Apr 08 04:23:37 ryzen kernel: amdgpu 0000:05:00.0: [drm] *ERROR* flip_done timed out
Apr 08 04:23:37 ryzen kernel: amdgpu 0000:05:00.0: [drm] *ERROR* [CRTC:55:crtc-1] commit wait timed out

At this point, the monitor fails to power on and the system hangs as soon as I open KDE "Display Settings". If I reboot, the system hangs immediately on logon as KDE's KScreen attempts to reapply 800x600 line-doubled to 1200p.

From my debugging, it seems that when modern amdgpu is given a DoubleScan mode, it sets up the GPU with the wrong register contents, causing the GPU to never properly output a frame, then crashes waiting for the GPU "pipe" (pipeline?) to finish processing.

Resolution

Personally I'd remove DoubleScan modes altogether from amdgpu/DC, since they're practically useless. I don't know of anyone (not even me who has written my own modelines with interlacing) who deliberately uses DoubleScan modes today, and you can replicate similar functionality if you truly desire by using xrandr to upscale a low resolution.

Alternatively you could properly implement DoubleScan to not hang, but this may be more work to fix the code, and will cause KDE to improperly select a 1200p instead of 600p resolution when picking 800x600 on a CRT monitor.

System specs

On my system, I believe DC is enabled, and my GPU (RX 570) uses DCE, according to drm/amd#1636 (comment 1566589).

Operating System: Arch Linux 
KDE Plasma Version: 5.27.3
KDE Frameworks Version: 5.104.0
Qt Version: 5.15.8
Kernel Version: 6.2.9-zen1-1-zen (64-bit)
Graphics Platform: X11
Processors: 12 × AMD Ryzen 5 5600X 6-Core Processor
Memory: 15.5 GiB of RAM
Graphics Processor: AMD Radeon RX 570 Series
Manufacturer: Gigabyte Technology Co., Ltd.
Product Name: B550M DS3H

Reported to KDE at https://bugs.kde.org/show_bug.cgi?id=464782.