6.9 regression: suspend not reaching deepest state on Asus Zenbook UM425IA (Ryzen 5 4500U)

Brief summary of the problem:

It appears that there is a s2idle regression in 6.9. kernel. This is an actual problem since the battery does drain very quickly. I have already found the laptop turned off with empty battery twice, which led me to investigate.

Log with 6.9.4:

Jun 18 09:25:07 kernel: PM: suspend entry (s2idle)
Jun 18 09:25:07 kernel: Filesystems sync: 0.031 seconds
Jun 18 12:42:20 kernel: Freezing user space processes
Jun 18 12:42:20 kernel: Freezing user space processes completed (elapsed 0.004 seconds)
Jun 18 12:42:20 kernel: OOM killer disabled.
Jun 18 12:42:20 kernel: Freezing remaining freezable tasks
Jun 18 12:42:20 kernel: Freezing remaining freezable tasks completed (elapsed 4.809 seconds)
Jun 18 12:42:20 kernel: printk: Suspending console(s) (use no_console_suspend to debug)
Jun 18 12:42:20 kernel: PM: suspend devices took 0.046 seconds
Jun 18 12:42:20 kernel: ACPI: EC: interrupt blocked
Jun 18 12:42:20 kernel: amd_pmc AMDI0005:00: Last suspend didn't reach deepest state
Jun 18 12:42:20 kernel: ACPI: EC: interrupt unblocked
Jun 18 12:42:20 kernel: [drm] PCIE GART of 1024M enabled.
Jun 18 12:42:20 kernel: [drm] PTB located at 0x000000F41FC00000
Jun 18 12:42:20 kernel: amdgpu 0000:04:00.0: amdgpu: SMU is resuming...
Jun 18 12:42:20 kernel: amdgpu 0000:04:00.0: amdgpu: dpm has been disabled
Jun 18 12:42:20 kernel: amdgpu 0000:04:00.0: amdgpu: SMU is resumed successfully!
Jun 18 12:42:20 kernel: nvme nvme0: 6/0/0 default/read/poll queues
Jun 18 12:42:20 kernel: [drm] VCN decode and encode initialized successfully(under DPG Mode).
Jun 18 12:42:20 kernel: [drm] JPEG decode initialized successfully.
Jun 18 12:42:20 kernel: [drm] JPEG decode initialized successfully.
Jun 18 12:42:20 kernel: amdgpu 0000:04:00.0: amdgpu: ring gfx uses VM inv eng 0 on hub 0
Jun 18 12:42:20 kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 1 on hub 0
Jun 18 12:42:20 kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 4 on hub 0
Jun 18 12:42:20 kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 5 on hub 0
Jun 18 12:42:20 kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 6 on hub 0
Jun 18 12:42:20 kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 7 on hub 0
Jun 18 12:42:20 kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 8 on hub 0
Jun 18 12:42:20 kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 9 on hub 0
Jun 18 12:42:20 kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 10 on hub 0
Jun 18 12:42:20 kernel: amdgpu 0000:04:00.0: amdgpu: ring kiq_0.2.1.0 uses VM inv eng 11 on hub 0
Jun 18 12:42:20 kernel: amdgpu 0000:04:00.0: amdgpu: ring sdma0 uses VM inv eng 0 on hub 8
Jun 18 12:42:20 kernel: amdgpu 0000:04:00.0: amdgpu: ring vcn_dec uses VM inv eng 1 on hub 8
Jun 18 12:42:20 kernel: amdgpu 0000:04:00.0: amdgpu: ring vcn_enc0 uses VM inv eng 4 on hub 8
Jun 18 12:42:20 kernel: amdgpu 0000:04:00.0: amdgpu: ring vcn_enc1 uses VM inv eng 5 on hub 8
Jun 18 12:42:20 kernel: amdgpu 0000:04:00.0: amdgpu: ring jpeg_dec uses VM inv eng 6 on hub 8
Jun 18 12:42:20 kernel: PM: resume devices took 0.275 seconds
Jun 18 12:42:20 kernel: OOM killer enabled.
Jun 18 12:42:20 kernel: Restarting tasks ... done.
Jun 18 12:42:20 kernel: random: crng reseeded on system resumption
Jun 18 12:42:20 kernel: ata1: SATA link down (SStatus 0 SControl 300)
Jun 18 12:42:20 kernel: ata2: SATA link down (SStatus 0 SControl 300)
Jun 18 12:42:20 kernel: PM: suspend exit

With 6.8.12:

Jun 10 21:20:27 kernel: Filesystems sync: 0.074 seconds
Jun 11 08:35:31 kernel: Freezing user space processes
Jun 11 08:35:31 kernel: Freezing user space processes completed (elapsed 0.003 seconds)
Jun 11 08:35:31 kernel: OOM killer disabled.
Jun 11 08:35:31 kernel: Freezing remaining freezable tasks
Jun 11 08:35:31 kernel: Freezing remaining freezable tasks completed (elapsed 0.001 seconds)
Jun 11 08:35:31 kernel: printk: Suspending console(s) (use no_console_suspend to debug)
Jun 11 08:35:31 kernel: PM: suspend devices took 0.029 seconds
Jun 11 08:35:31 kernel: ACPI: EC: interrupt blocked
Jun 11 08:35:31 kernel: ACPI: EC: interrupt unblocked
Jun 11 08:35:31 kernel: [drm] PCIE GART of 1024M enabled.
Jun 11 08:35:31 kernel: [drm] PTB located at 0x000000F41FC00000
Jun 11 08:35:31 kernel: amdgpu 0000:04:00.0: amdgpu: SMU is resuming...
Jun 11 08:35:31 kernel: amdgpu 0000:04:00.0: amdgpu: dpm has been disabled
Jun 11 08:35:31 kernel: amdgpu 0000:04:00.0: amdgpu: SMU is resumed successfully!
Jun 11 08:35:31 kernel: [drm] VCN decode and encode initialized successfully(under DPG Mode).
Jun 11 08:35:31 kernel: [drm] JPEG decode initialized successfully.
Jun 11 08:35:31 kernel: amdgpu 0000:04:00.0: amdgpu: ring gfx uses VM inv eng 0 on hub 0
Jun 11 08:35:31 kernel: amdgpu 0000:04:00.0: amdgpu: ring gfx_low uses VM inv eng 1 on hub 0
Jun 11 08:35:31 kernel: amdgpu 0000:04:00.0: amdgpu: ring gfx_high uses VM inv eng 4 on hub 0
Jun 11 08:35:31 kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 5 on hub 0
Jun 11 08:35:31 kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 6 on hub 0
Jun 11 08:35:31 kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 7 on hub 0
Jun 11 08:35:31 kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 8 on hub 0
Jun 11 08:35:31 kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 9 on hub 0
Jun 11 08:35:31 kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 10 on hub 0
Jun 11 08:35:31 kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 11 on hub 0
Jun 11 08:35:31 kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 12 on hub 0
Jun 11 08:35:31 kernel: amdgpu 0000:04:00.0: amdgpu: ring kiq_0.2.1.0 uses VM inv eng 13 on hub 0
Jun 11 08:35:31 kernel: amdgpu 0000:04:00.0: amdgpu: ring sdma0 uses VM inv eng 0 on hub 8
Jun 11 08:35:31 kernel: amdgpu 0000:04:00.0: amdgpu: ring vcn_dec uses VM inv eng 1 on hub 8
Jun 11 08:35:31 kernel: amdgpu 0000:04:00.0: amdgpu: ring vcn_enc0 uses VM inv eng 4 on hub 8
Jun 11 08:35:31 kernel: amdgpu 0000:04:00.0: amdgpu: ring vcn_enc1 uses VM inv eng 5 on hub 8
Jun 11 08:35:31 kernel: amdgpu 0000:04:00.0: amdgpu: ring jpeg_dec uses VM inv eng 6 on hub 8
Jun 11 08:35:31 kernel: nvme nvme0: 8/0/0 default/read/poll queues
Jun 11 08:35:31 kernel: PM: resume devices took 0.255 seconds
Jun 11 08:35:31 kernel: OOM killer enabled.
Jun 11 08:35:31 kernel: Restarting tasks ... done.
Jun 11 08:35:31 kernel: random: crng reseeded on system resumption
Jun 11 08:35:31 kernel: PM: suspend exit

Hardware description:

• amd_s2idle report: s2idle_report-2024-06-18.txt. SMT requirement is a problem since this CPU does not support it at all.

System information:

• Distro name and Version: Fedora 40
• Kernel version: Linux snowball3 6.9.4-200.fc40.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Jun 12 13:33:34 UTC 2024 x86_64 GNU/Linux

How to reproduce the issue:

Upgrade Fedora 40 to 6.9 kernel. Suspend

Attached files:

changed the description

added Renoir s0ix labels

Can you bisect?

Not in the short term I'm afraid. Last time I tried this with Fedora kernels it took a lot of effort due to how Fedora kernel git is built and due to having to always build from scratch instead of rebuilding changed files. I have one more child now so time is even more scarce.

I can see how far I can get using built kernels from koji.

The regression seems to be from kernel-6.9.0-0.rc0.20240314git480e035fc4c7.5 to kernel-6.9.0-0.rc0.20240315gite5eb28f6d1af.7. Not sure how useful this is, but going any further requires that I do rebuilds myself.

Assuming I got this right git log --oneline 480e035fc4..e5eb28f6 is a pretty big list. The most suspicious to me would be these:

6d4266675279 PCI/ASPM: Update save_state when configuration changes
64dbb2d70744 PCI/ASPM: Disable L1 before configuring L1 Substates
c198fafa0125 PCI/ASPM: Call pci_save_ltr_state() from pci_save_pcie_state()
17423360a27a PCI/ASPM: Save L1 PM Substates Capability for suspend/resume

I have managed to make the time to run the full bisect and it looks like

0e02ca29a563 lib/sort: optimize heapsort with double-pop variation

is the culprit. I will revert it on top of current 6.9.x just to be sure.

Confirmed with 6.9.5: reverting 0e02ca29a563 fixes the regression.

Please report the regression to the patch author and LKML.

Done. I have set you in CC, please feel free to inform whoever else might be interested.

Drop the link to it here for anyone else affected.

CC @kernels_regtracker

https://lore.kernel.org/lkml/70674dc7-5586-4183-8953-8095567e73df@gmail.com/

Can you please run the report with --force to get around the SMT requirement? I want to see debugging data that it captures when launched with the script explicitly. I'll need to adjust the script for designs without SMT. The ones that have it; if you turn it off cause problems.

added bisected label and removed Renoir s0ix labels

added Renoir s0ix labels