admgpu aspm cause hang during suspend&resume with wx3200 on adl platform
Brief summary of the problem:
Use wx3200 as a video output, during suspend/resume, system would hang.
Hardware description:
- CPU: i7-12700k
- GPU: wx3200
- System Memory:
- Display(s): wx3200
- Type of Display Connection: hdmi
System information:
- Distro name and Version: Ubuntu 20.04.1
- Kernel version: 5.14/5.17-rc2
- Custom kernel: ubuntu-5.14-oem-1021
- AMD official driver version:
How to reproduce the issue:
use wx3200 as the video output.
Trigger suspend and the system would hang.
Found it's a regression caused by
0064b0ce85) drm/amd/pm: enable ASPM by default
With v5.14 and v5.17rc2 and disable ampgpu aspm, would not get the issue.
Attached files:
Screenshots/video files
Log files (for system lockups / game freezes / crashes)
- Dmesg log
dmesg_514_20220128
dmesg_517rc2_mainline_20220128 - lspci
lspci_tnv_20220128
Activity
Does this patch help? #1723 (comment 1241120)
@agd5f yes, the patch helps to resolve the suspend issue with wx3200.
but why is only wx3200 affected by aspm? Tried W6600 and didn't find the issue.
Are wx3200 and w6600 different series?
Thanks,hi @superm1 ,
you could get my
tested-by:koba.ko@canonical.com
mentioned in issue #1723 (closed)
That patch was submitted for code review here: https://patchwork.freedesktop.org/patch/474615/ it should flow up for 5.18 (or maybe 5.17-rcX if there is still time as it's a fix).
@superm1 I checked it again, With the same machine and wx3200, i still reproduced the issue with the patch.
In Ubuntu kernel, aspm is enabled so amdgpu_aspm would be as default(-1). that's the same configuration before apply patch. issue could still be triggered.
With the same kernel and force amdgpu.aspm=0, the issue would be gone.
@koba I thought you tested successfully before? That's the main reason we merged it. What changed from your previous test?
tested two different machines and got passed before. I just restored machine as default and install the proposed kernel.
Edited by kobakoi didn't move cards between pcie ports and only disable the secure-boot(it's the same before restored).
i verified both on oem kernel and mainline kernel.
Let me check mainline kernel again.
Update: with the mainline kernel(20220303), still reproduce issue.Edited by kobakodid you maybe have any debugging parameters on your kernel command line? can you reference your journal from the boots that it worked? Maybe it's a regression between the kernel you put the patch on top of and the latest mainline (along with something that backported to your OEM tree since then too)?
@superm1 currently the target is a clear env because i have reset to default. There's no any additional options in the grub. I reset the mainline branch with "-hard" and there's no any others modifications on it.
I just don't get the difference you're seeing. IOW is it possible something went upstream between that commit and now and came back to -stable (and your OEM tree) that caused it to not work? What if you reset it to what commit you were at previously that you applied the patch and was working?
@superm1 no idea, before sru, i always reset the repo with "-hard" and apply the patch then verify.
Can you check what commit hash you built that kernel on when you tested?
The reason I'm fixating on this is that ASPM decision should come from outside GPU driver. So this might be better to live in PCI quirks or in BIOS. But if you have proof that it worked without either of those at some time we should find out "why" that stopped working to revert that commit.
@superm1 ok, i will bisect it.
"acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration"
This means OS doesn't touch any ASPM functionality. Any change in BIOS configuration (reset to defaults, BIOS update etc.)?
/* * Disabling ASPM is intended to prevent the kernel from modifying * existing hardware state, not to clear existing state. To that end: * (a) set policy to POLICY_DEFAULT in order to avoid changing state * (b) prevent userspace from changing policy */Edited by lijo lazar@superm1 i can't find any clue
Maybe i change something in BIOS but forget.Does BIOS have an option to turn on/off ASPM? Maybe it's misconfigured in the first place. We could do stuff like
intel_core_rkl_chkin the GPU driver, but what's the right criteria to match? Do all ALD systems fail? Or just this model from this manufacturer?Edited by Mario Limonciello
mentioned in commit agd5f/linux@c9574ad7
mentioned in commit agd5f/linux@cba07cce
mentioned in commit agd5f/linux@4397d09c
mentioned in commit agd5f/linux@7294863a
mentioned in issue #1955 (closed)
Discussing the patch [PATCHv2] drm/amdgpu: disable ASPM on Intel AlderLake based systems, could the current state of the issue please be summarized? With the positive and negative test result, it is just too confusing.
[ 0.000000] Linux version 5.15.0-051500-generic (kobako@barbatos) (gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #202110312130 SMP PREEMPT Fri Jan 28 13:03:24 CST 2022 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.0-051500-generic root=UUID=f33c77fa-74c6-49bf-807c-e6828e535d0f ro drm.debug=0x1ff log_buf_len=16m "dyndbg=file pci-driver.c +p;file pci.c +p" [ 0.000000] DMI: Dell Inc. Precision 3660/, BIOS 0.14.81 11/08/2021 […] [ 2.228238] [drm:check_atom_bios [amdgpu]] ATOMBIOS detected [ 2.228338] amdgpu 0000:01:00.0: amdgpu: Fetched VBIOS from VFCT [ 2.228339] amdgpu: ATOM BIOS: 113-D0155300-100 […]Card:
0000:01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Lexa XT [Radeon PRO WX 3200] (rev 10) (prog-if 00 [VGA controller]) Subsystem: Dell Lexa XT [Radeon PRO WX 3200]@koba, where does it freeze exactly? How did you acquire the Linux logs? Where you able to log in over the network, on some other console like serial console?
Somehow the verbose log messages should give some hint, what is going wrong, shouldn’t they?
I saw:
[ 58.101385] Freezing of tasks failed after 20.003 seconds (4 tasks refusing to freeze, wq_busy=0): [ 58.101448] task:tlp state:D stack: 0 pid: 1471 ppid: 752 flags:0x00004004 [ 58.101460] Call Trace: [ 58.101465] <TASK> [ 58.101473] __schedule+0x2af/0x7d0 [ 58.101488] schedule+0x55/0xd0 [ 58.101495] schedule_timeout+0x200/0x290 [ 58.101505] wait_for_completion+0x8b/0xf0 [ 58.101513] ? watchdog_disable+0x50/0x50 [ 58.101520] smp_call_on_cpu+0xc5/0x100 [ 58.101530] ? wake_up_all_idle_cpus+0x70/0x70 [ 58.101539] ? watchdog_disable+0x50/0x50 [ 58.101546] lockup_detector_reconfigure+0x3a/0x140 [ 58.101553] proc_watchdog_update+0x2a/0x30 [ 58.101559] proc_watchdog_common+0xa7/0xb0 [ 58.101566] proc_nmi_watchdog+0x2f/0x40 [ 58.101572] proc_sys_call_handler+0x16d/0x250 [ 58.101581] proc_sys_write+0x13/0x20 [ 58.101586] new_sync_write+0x117/0x1b0 [ 58.101596] vfs_write+0x1a7/0x290 [ 58.101602] ksys_write+0x67/0xe0 [ 58.101609] __x64_sys_write+0x1a/0x20 [ 58.101615] do_syscall_64+0x59/0xc0 [ 58.101622] ? asm_exc_page_fault+0x8/0x30 [ 58.101628] entry_SYSCALL_64_after_hwframe+0x44/0xae [ 58.101634] RIP: 0033:0x7f0cb1f2a1e7 [ 58.101641] RSP: 002b:00007ffec48a0338 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 58.101649] RAX: ffffffffffffffda RBX: 00007f0cb2eb7070 RCX: 00007f0cb1f2a1e7 [ 58.101653] RDX: 0000000000000002 RSI: 00007f0cb2eb7070 RDI: 0000000000000001 [ 58.101657] RBP: 0000000000000002 R08: 00007f0cb2eb6d84 R09: 00007ffec48a00b0 [ 58.101660] R10: 00007f0cb2eb6d82 R11: 0000000000000246 R12: 0000000000000001 [ 58.101663] R13: 0000000000000002 R14: 7fffffffffffffff R15: 00007ffec48a0500 [ 58.101670] </TASK> [ 58.101698] task:gdbus state:D stack: 0 pid: 2073 ppid: 1544 flags:0x00004006 [ 58.101706] Call Trace: [ 58.101708] <TASK> [ 58.101711] __schedule+0x2af/0x7d0 [ 58.101719] schedule+0x55/0xd0 [ 58.101725] schedule_timeout+0x200/0x290 [ 58.101733] ? sched_clock_cpu+0xe/0xc0 [ 58.101741] ? raw_spin_rq_unlock+0x10/0x30 [ 58.101750] wait_for_completion+0x8b/0xf0 [ 58.101758] __flush_work+0x137/0x1f0 [ 58.101765] ? flush_workqueue_prep_pwqs+0x120/0x120 [ 58.101772] flush_delayed_work+0x33/0x50 [ 58.101778] fsnotify_wait_marks_destroyed+0x15/0x20 [ 58.101787] fsnotify_destroy_group+0x5d/0xf0 [ 58.101794] ? __call_rcu+0xac/0x270 [ 58.101800] inotify_release+0x1c/0x50 [ 58.101807] __fput+0x9c/0x260 [ 58.101814] ____fput+0xe/0x10 [ 58.101820] task_work_run+0x64/0xa0 [ 58.101827] do_exit+0x349/0xb10 [ 58.101837] do_group_exit+0x35/0xa0 [ 58.101845] get_signal+0x121/0x9c0 [ 58.101852] ? exit_to_user_mode_prepare+0x3d/0x1c0 [ 58.101860] arch_do_signal_or_restart+0xf3/0x2c0 [ 58.101866] ? fput+0x13/0x20 [ 58.101872] ? __sys_sendmsg+0x97/0xb0 [ 58.101880] exit_to_user_mode_prepare+0x12c/0x1c0 [ 58.101886] syscall_exit_to_user_mode+0x27/0x50 [ 58.101894] do_syscall_64+0x69/0xc0 [ 58.101900] ? asm_sysvec_reschedule_ipi+0xa/0x20 [ 58.101906] entry_SYSCALL_64_after_hwframe+0x44/0xae [ 58.101911] RIP: 0033:0x7fb13c228aff [ 58.101914] RSP: 002b:00007fb137de4aa0 EFLAGS: 00000293 ORIG_RAX: 0000000000000007 [ 58.101919] RAX: fffffffffffffdfc RBX: 00007fb13d3470d0 RCX: 00007fb13c228aff [ 58.101922] RDX: 00000000ffffffff RSI: 0000000000000004 RDI: 00007fb11800a3b0 [ 58.101926] RBP: 00007fb11800a3b0 R08: 0000000000000000 R09: 0000000000000001 [ 58.101929] R10: 00007fb128017980 R11: 0000000000000293 R12: 0000000000000004 [ 58.101932] R13: 00007fb137de4ae4 R14: 00000000ffffffff R15: 00007fb13eb82da0 [ 58.101939] </TASK> [ 58.101955] task:single state:D stack: 0 pid: 2362 ppid: 1544 flags:0x00004004 [ 58.101960] Call Trace: [ 58.101963] <TASK> [ 58.101966] __schedule+0x2af/0x7d0 [ 58.101973] schedule+0x55/0xd0 [ 58.101979] schedule_timeout+0x200/0x290 [ 58.101987] ? raw_spin_rq_unlock+0x10/0x30 [ 58.101994] ? finish_task_switch+0xac/0x270 [ 58.102004] wait_for_completion+0x8b/0xf0 [ 58.102011] __flush_work+0x137/0x1f0 [ 58.102017] ? flush_workqueue_prep_pwqs+0x120/0x120 [ 58.102024] flush_work+0x10/0x20 [ 58.102029] lru_add_drain_all+0x15b/0x1e0 [ 58.102040] generic_fadvise+0x216/0x270 [ 58.102048] vfs_fadvise+0x25/0x30 [ 58.102053] ksys_fadvise64_64+0x41/0x80 [ 58.102059] ? syscall_trace_enter.isra.0+0xa0/0x1d0 [ 58.102066] __x64_sys_fadvise64+0x1e/0x30 [ 58.102072] do_syscall_64+0x59/0xc0 [ 58.102077] ? syscall_exit_to_user_mode+0x27/0x50 [ 58.102086] ? __x64_sys_mremap+0x25/0x30 [ 58.102094] ? do_syscall_64+0x69/0xc0 [ 58.102100] ? exit_to_user_mode_prepare+0x3d/0x1c0 [ 58.102104] ? do_user_addr_fault+0x1dd/0x650 [ 58.102114] ? irqentry_exit_to_user_mode+0x9/0x20 [ 58.102122] ? irqentry_exit+0x33/0x40 [ 58.102130] ? exc_page_fault+0x89/0x180 [ 58.102137] ? asm_exc_page_fault+0x8/0x30 [ 58.102143] entry_SYSCALL_64_after_hwframe+0x44/0xae [ 58.102148] RIP: 0033:0x7fa50796cc4e [ 58.102151] RSP: 002b:00007fa4c6cfac48 EFLAGS: 00000246 ORIG_RAX: 00000000000000dd [ 58.102156] RAX: ffffffffffffffda RBX: 00007fa4c6cfac80 RCX: 00007fa50796cc4e [ 58.102159] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000000000f [ 58.102162] RBP: 00007fa4bc011bd0 R08: 00007fa4c6126000 R09: 00007fa4c6327000 [ 58.102165] R10: 0000000000000004 R11: 0000000000000246 R12: 00007fa509071360 [ 58.102168] R13: 000000000000ffff R14: 00007fa4c6cfac50 R15: 000000000000000f [ 58.102174] </TASK> [ 58.102177] task:tlp state:D stack: 0 pid: 2371 ppid: 1 flags:0x00000004 [ 58.102184] Call Trace: [ 58.102185] <TASK> [ 58.102188] __schedule+0x2af/0x7d0 [ 58.102196] schedule+0x55/0xd0 [ 58.102202] schedule_preempt_disabled+0x15/0x20 [ 58.102210] __mutex_lock.isra.0+0x215/0x480 [ 58.102219] __mutex_lock_slowpath+0x13/0x20 [ 58.102226] mutex_lock+0x32/0x40 [ 58.102233] proc_watchdog_common+0x3a/0xb0 [ 58.102240] proc_nmi_watchdog+0x2f/0x40 [ 58.102247] proc_sys_call_handler+0x16d/0x250 [ 58.102254] proc_sys_write+0x13/0x20 [ 58.102259] new_sync_write+0x117/0x1b0 [ 58.102267] vfs_write+0x1a7/0x290 [ 58.102274] ksys_write+0x67/0xe0 [ 58.102281] __x64_sys_write+0x1a/0x20 [ 58.102287] do_syscall_64+0x59/0xc0 [ 58.102292] ? filp_close+0x60/0x70 [ 58.102301] ? syscall_exit_to_user_mode+0x27/0x50 [ 58.102309] ? __x64_sys_close+0x12/0x40 [ 58.102317] ? do_syscall_64+0x69/0xc0 [ 58.102323] entry_SYSCALL_64_after_hwframe+0x44/0xae [ 58.102327] RIP: 0033:0x7f6a8a2dd1e7 [ 58.102330] RSP: 002b:00007fffce039258 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 58.102334] RAX: ffffffffffffffda RBX: 00007f6a8b396e30 RCX: 00007f6a8a2dd1e7 [ 58.102337] RDX: 0000000000000002 RSI: 00007f6a8b396e30 RDI: 0000000000000001 [ 58.102340] RBP: 0000000000000002 R08: 00007f6a8b396d64 R09: 00007fffce038fd0 [ 58.102343] R10: 00007f6a8b396d62 R11: 0000000000000246 R12: 0000000000000001 [ 58.102346] R13: 0000000000000002 R14: 7fffffffffffffff R15: 00007fffce039420 [ 58.102352] </TASK>@koba, is that also present with the other card? @koba, please also try the current Linux 5.18-rc2. Please also attach the logs from that try.
mentioned in commit agd5f/linux@acf7b89a
mentioned in commit agd5f/linux@e02fe3bc
mentioned in commit agd5f/linux@aa482ddc
This change to avoid ASPM on ALD is landed as https://github.com/torvalds/linux/commit/aa482ddca85a3485be0e7b83a0789dc4d987670b (which is also CC stable), closing issue.
mentioned in commit nouveau@7a9e13b8
