Cannot disable wakeup sources on Yellow Carp / Rembrandt 6850U w/ kernels 6.0 and 6.1

added s0ix label

In particular, if the screen presses down on the trackpoint (which happens fairly frequently with the lid closed), the machine will wake up from s2idle and will eventually overheat or run out of battery.

This is a bit odd to me - it should wake up but eventually go back to sleep because the lid is closed and it stays idle a while, shouldn't it? Or do you have that turned off in your DE?

2023-02-10 08:04:25,865 DEBUG:	WCN6855 version string: ath11k_pci 0000:02:00.0: fw_version 0x11090c35 fw_build_timestamp 2022-04-18 20:23 fw_build_id WLAN.HSP.1.1-03125-QCAHSPSWPL_V1_V2_SILICONZ_LITE-3.6510.9
2023-02-10 08:04:25,865 ERROR:	❌ WCN6855 WLAN may cause spurious wakeups (fw version 0x11090c35)

I think this might actually be playing into your problem. Can you please upgrade this firmware as advised by the script and see if you can reproduce it still?

Aaaaah, okay. Yeah, I tend to run my laptops with the lid closed at times, so I disabled suspend with the lid closed. /etc/systemd/logind.conf

My general approach is that I will intentionally instruct the laptop when to suspend, and I will intentionally bring it out of suspend.

I think my broader confusion here is why disabling wakeup sources seems to be a no-op on my machine.

Aaaaah, okay. Yeah, I tend to run my laptops with the lid closed at times, so I disabled suspend with the lid closed. /etc/systemd/logind.conf My general approach is that I will intentionally instruct the laptop when to suspend, and I will intentionally bring it out of suspend.

Got it.

I think my broader confusion here is why disabling wakeup sources seems to be a no-op on my machine.

Well two things.

That bug with WCN6855 causes weird things to happen at wakeup. For example, normally unplugging the power adapter should cause the system to leave HW sleep, notify the kernel that it's not plugged in and kernel goes back to HW sleep. Because of that WCN6855 bug it wakes up fully. I strongly suspect this is the root cause of your problems.
You should be disabling wakeups from sysfs for individual devices. That will make the kernel not mark IRQs for wakeup. For example trackpoint you should disable in /sys/bus/serio/devices/serio1/power/wakeup.

Okay, I will look into the WCN6855 firmware update; thanks!

The trackpoint wakeup setting defaults to disabled:

$cat /sys/bus/serio/devices/serio1/power/wakeup
disabled

And the keyboard one starts off enabled, but even after I disable it, pressing any key on the keyboard results in a full wakeup

$cat /sys/bus/serio/devices/serio0/power/wakeup
enabled

OK, let's see whether those values are still non-functional with WCN6855 firmware updated. If so, what I want to see is the following:

Use the duration argument for amd_s2idle.py to suspend the machine with the lid open for a minute or so.
Move the trackpoint (which should be disabled by default as you said above).
Upload the new report generated from the script.

marked this issue as related to #2377 (closed)

#cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-6.1.0-2-amd64 root=/dev/mapper/carbon--vg-root ro rootflags=subvol=@rootfs quiet acpi.ec_no_wakeup=1

#dmesg | grep -B1 -A4 SILICONZ
[   12.298553] ath11k_pci 0000:02:00.0: chip_id 0x12 chip_family 0xb board_id 0xff soc_id 0x400c1211
[   12.298561] ath11k_pci 0000:02:00.0: fw_version 0x110b196e fw_build_timestamp 2022-12-22 12:54 fw_build_id WLAN.HSP.1.1-03125-QCAHSPSWPL_V1_V2_SILICONZ_LITE-3.6510.23
[   12.305727] ath11k_pci 0000:02:00.0: firmware: direct-loading firmware ath11k/WCN6855/hw2.1/board-2.bin
[   12.306132] ath11k_pci 0000:02:00.0: firmware: direct-loading firmware ath11k/WCN6855/hw2.1/regdb.bin
[   12.313628] ath11k_pci 0000:02:00.0: firmware: direct-loading firmware ath11k/WCN6855/hw2.1/board-2.bin
[   12.339913] ath11k_pci 0000:02:00.0: firmware: direct-loading firmware ath11k/WCN6855/hw2.1/m3.bin

#cat /sys/bus/serio/devices/serio0/power/wakeup
enabled

#cat /sys/bus/serio/devices/serio1/power/wakeup
disabled

#cat /proc/acpi/wakeup 
Device	S-state	  Status   Sysfs node
GPP5	  S4	*disabled  pci:0000:00:02.1
GPP6	  S0	*disabled  pci:0000:00:02.2
GPP7	  S0	*disabled
GP11	  S0	*disabled
SWUS	  S4	*disabled
GP12	  S0	*disabled
SWUS	  S4	*disabled
XHC0	  S3	*disabled  pci:0000:04:00.3
XHC1	  S3	*disabled  pci:0000:04:00.4
NHI0	  S0	*disabled
NHI1	  S0	*disabled
XHC2	  S3	*disabled  pci:0000:05:00.0
XHC3	  S3	*disabled  pci:0000:05:00.3
XHC4	  S3	*disabled  pci:0000:05:00.4
LID	  S4	*disabled  platform:PNP0C0D:00
SLPB	  S3	*disabled  platform:PNP0C0E:00

Set the deadline for 60 seconds, and poked the trackpoint at 20 seconds, at which point the laptop woke up.
s2idle_report.txt

Can you please repeat with secure boot turned off? There are some logging details that can be enabled from dynamic debugging that I want to see regarding what gpio wakes the system in your case.

Reran with SecureBoot disabled: s2idle_report.txt

2023-02-11 17:58:00,267 INFO:   ○ Kernel suspended for total of 18.4540 seconds
2023-02-11 17:58:00,267 INFO:   ○ Suspend count: 1
2023-02-11 17:58:00,268 INFO:   ○ Hardware sleep cycle count: 1
2023-02-11 17:58:00,268 INFO:   ○ GPIOs active: ['0']
2023-02-11 17:58:00,268 INFO:   ○ Wakeups triggered from IRQs: [7, 9]
2023-02-11 17:58:00,268 INFO:   ○ Used Microsoft uPEP GUID in LPS0 _DSM
2023-02-11 17:58:00,268 INFO:   ○ Woke up from IRQ 7 (IR-IO-APIC 7-fasteoi pinctrl_amd)
2023-02-11 17:58:00,268 INFO:   ○ Spent 18.898193 seconds in a hardware sleep state

and in case /proc/interrupts shows any new data with lockdown disabled: /proc/interrupts.txt

This was the second attempt this boot cycle. The first attempt woke up immediately, but I'm guessing I held down a key to long as the system was going into suspend (switched from sudo ../amd_s2idle.py --duration 60 to sleep 2; sudo ../amd_s2idle.py --duration 60 for the subsequent run, shown above)
immediate wakeup s2idle_report.txt

Setup is the same:

#cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-6.1.0-2-amd64 root=/dev/mapper/carbon--vg-root ro rootflags=subvol=@rootfs quiet acpi.ec_no_wakeup=1

#cat /sys/bus/serio/devices/serio0/power/wakeup
enabled

#cat /sys/bus/serio/devices/serio1/power/wakeup
disabled

#awk '/enabled/ {print $1}' wakeup > wakeup; cat wakeup 
Device	S-state	  Status   Sysfs node
GPP5	  S4	*disabled  pci:0000:00:02.1
GPP6	  S0	*disabled  pci:0000:00:02.2
GPP7	  S0	*disabled
GP11	  S0	*disabled
SWUS	  S4	*disabled
GP12	  S0	*disabled
SWUS	  S4	*disabled
XHC0	  S3	*disabled  pci:0000:04:00.3
XHC1	  S3	*disabled  pci:0000:04:00.4
NHI0	  S0	*disabled
NHI1	  S0	*disabled
XHC2	  S3	*disabled  pci:0000:05:00.0
XHC3	  S3	*disabled  pci:0000:05:00.3
XHC4	  S3	*disabled  pci:0000:05:00.4
LID	  S4	*disabled  platform:PNP0C0D:00
SLPB	  S3	*disabled  platform:PNP0C0E:00

Reran with SecureBoot disabled: s2idle_report.txt

So GPIO0, which is what Lenovo uses for notifying all ACPI related events like a power button, lid, or EC. You can see in the debug logs that _EVT case for 0x00 handles this. I suspect that it's going down the power button path. Can you run acpi_listen in another terminal while you reproduce? If you end up seeing a power button event I would say this looks like probably a BIOS bug that they're not handling this event properly.

If it's not a power button event we'll need to turn on some more verbose ACPI event layers to determine why this is happening and if this is a kernel bug or BIOS bug. Are you familiar with doing that?

immediate wakeup s2idle_report.txt

In this case you woke up from GPIO 18, which is associated with GPP3 on your machine. The script doesn't currently capture what that's associated with (something I might improve in the script for the future). Do you know what that matches on your machine?

something I might improve in the script for the future

I've updated the script to capture this mapping. If you don't know what GPP3 is on your machine, you can upgrade to that we can see.

Okay, ran a couple more experiments. Sadly, it sounds like your hypothesis is confirmed that the Lenovo BIOS is sending keyboard and trackpoint wakeups as power-button events. I also had a few more instances of the unexpected wakeup (interestingly, I never see these unexpected wakeups when suspending the system via the "Suspend" button in the UI).

I tried some events without suspending, and here's what they looked like in acpi_listen: acpi_listen_events.txt

Now, the s2idle reports and (for some) matching acpi_listen logs for multiple suspend attempts:

s2idle_report-2023-02-23.0_woke_immediately.txt
s2idle_report-2023-02-23.1_woke_immediately.txt acpi_listen_events.1_woke_immediately.txt (I unplugged mains power after this run, since I kept seeing power-related ACPI events in acpi_listen)
s2idle_report-2023-02-23.2_woke_immediately_no_mains.txt (after this run, I removed my cordless USB mouse receiver, and the system started staying in suspend)
s2idle_report-2023-02-23.3_power_btn_wake.txt acpi_listen_events.3_power_btn_wake.txt
s2idle_report-2023-02-23.4_keyboard_wake.txt acpi_listen_events.4_keyboard_wake.txt
s2idle_report-2023-02-23.5_trackpoint_wake.txt acpi_listen_events.5_trackpoint_wake.txt
s2idle_report-2023-02-23.6_unexpected_wake.txt acpi_listen_events.6_unexpected_wake.txt
s2idle_report-2023-02-23.7_unexpected_wake.txt acpi_listen_events.7_unexpected_wake.txt
I clicked "Suspend" and woke the system up with the trackpoint this time: acpi_listen_events.8_trackpoint_wake.txt

System configuration during these experiments:

#cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-6.1.0-2-amd64 root=/dev/mapper/carbon--vg-root ro rootflags=subvol=@rootfs quiet acpi.ec_no_wakeup=1

#cat /proc/acpi/wakeup 
Device	S-state	  Status   Sysfs node
GPP5	  S4	*disabled  pci:0000:00:02.1
GPP6	  S0	*disabled  pci:0000:00:02.2
GPP7	  S0	*disabled
GP11	  S0	*disabled
SWUS	  S4	*disabled
GP12	  S0	*disabled
SWUS	  S4	*disabled
XHC0	  S3	*disabled  pci:0000:04:00.3
XHC1	  S3	*disabled  pci:0000:04:00.4
NHI0	  S0	*disabled
NHI1	  S0	*disabled
XHC2	  S3	*disabled  pci:0000:05:00.0
XHC3	  S3	*disabled  pci:0000:05:00.3
XHC4	  S3	*disabled  pci:0000:05:00.4
LID	  S4	*disabled  platform:PNP0C0D:00
SLPB	  S3	*disabled  platform:PNP0C0E:00

#cat /sys/bus/serio/devices/serio0/power/wakeup
enabled

#cat /sys/bus/serio/devices/serio1/power/wakeup
disabled

I also had a few more instances of the unexpected wakeup

This one and other immediate wake ones are waking from GPIO 18, which means the WLAN controller. Maybe their state machine isn't doing well with the constant attempts at suspend cycles.

If it's a common problem you should be able to ignore it with this on your kernel command line: acpi_mask_gpe=0x0e gpiolib_acpi.ignore_interrupt=AMDI0030:00@18 But keep in mind this breaks WoWLAN if you rely upon that at all.

Sadly, it sounds like your hypothesis is confirmed that the Lenovo BIOS is sending keyboard and trackpoint wakeups as power-button events

Yeah I think so too looking at your logs.

So for one, I'm guessing this will need to turn into some kind of bug report to Lenovo. How does that work?

Second, to get me by in the meanwhile, I think I'd like to modify my kernel so that it only wakes up on lid-open events. Where would I look to start making that change? (I'm pretty familiar with compiling my own kernels and I know C, so mostly trying to understand what files I should be looking in)

So for one, I'm guessing this will need to turn into some kind of bug report to Lenovo. How does that work?

I think you'll want to notify them through their forums. Don't expect a fast process though. Changing the BIOS is a lot of work and the risk of regression is high.

Cc @markpearson too.

Second, to get me by in the meanwhile, I think I'd like to modify my kernel so that it only wakes up on lid-open events. Where would I look to start making that change? (I'm pretty familiar with compiling my own kernels and I know C, so mostly trying to understand what files I should be looking in)

It might be easier to go the patched DSDT/SSDT. You can remove Notify() from _AEI events based on the criteria that you deem necessary.

Thanks Mario.

@xsdg - apologies if I've missed it - but which platform is this on? If it's on one of our Linux certified ones I'd like to reproduce it and can flag it to the FW team.

@mpearson ThinkPad T14 Gen 3 AMD (14") 21CFCTO1WWUS1

I seem to be having the same issue with my AMD 6850U-based T16 ThinkPad. (21CH000JUS)

I can reliably get it to wake from sleep by closing the lid and gently pressing about where the TrackPoint is, or by plugging in USB devices. Sometimes it just seems to wake up on it's own, though I didn't manage to catch it with amd_s2idle.py after a couple days.

Here are some amd_s2idle.py logs:

lid-pressed.txt - Closed and pressed the lid above the trackpoint.
plug-in.txt - Left the lid open and plugged in the power adapter.
1h-success.txt - Set the wakealarm to 1 hour and didn't touch it until it woke itself up.

IIRC plugging in USB devices I believe is an intended wakeup source.

As for the trackpoint causing the wakeup, I think this is something that will need to be filed with Lenovo and fixed by their EC.

added Rembrandt label

Cannot disable wakeup sources on Yellow Carp / Rembrandt 6850U w/ kernels 6.0 and 6.1

Brief summary of the problem:

Hardware description:

System information:

How to reproduce the issue:

Attached files:

Designs

Child items ...

Activity

Admin message

Admin message

Cannot disable wakeup sources on Yellow Carp / Rembrandt 6850U w/ kernels 6.0 and 6.1

Brief summary of the problem:

Hardware description:

System information:

How to reproduce the issue:

Attached files:

Activity