Suspend does not work with encrypted NVME

@superm1

5.14.0.rc7 is partially working on an IdeaPad 5 Pro 16ACH6.

I can suspend and resume if and only if i disable the NVME hardware encryption. dmesg_suspend_no_nvme_password.txt

But with a proper UEFI level (TCG OPAL i guess) encrypted drive it does not work because the NVME is put in such a deep sleep that the decrypted state is reset. Either the NVME password would have to be asked again as it is at System start from the UEFI or the drive should not be powered off completely to retain its decrypted state.

It is working perfectly fine with Windows 10. I do not even need to enter a password, the NVME stays decrypted or they somehow restore the decryption state without user intervention. I also have an Intel yoga slim 7i with hardware encrypted NVME and suspend is working there too with Linux. There is also no point to switch off an NVME completely, because in the lowest power states it draws only 2.5 milliwatt.

https://docs.microsoft.com/en-us/windows-hardware/design/device-experiences/modern-standby-sleepstudy

sleepstudy-report.html

mentioned in issue #1230 (closed)

Can you run the study for longer - 30 minutes or so? You can see in that graph at least in that duration of test (~12 minutes), it never went into HW DRIPS (SOC deepest state). So either that is indicating Windows is blocking it from HW drips (due to OPAL) or it didn't have enough time.

One more comment - if you can please try without any USB device plugged in on the Windows sleep study.

I am booting windows from an USB drive. I will start with 30 Minutes and if that does not help i can install it to the NVME.

So it seems that i have to really install to the internal NVME... I have to backup my data first, resize partitions and install Windows 10. This might take some time and i will report back with newly generated sleep data.

sleepstudy-report-usb.html

I installed Windows 10 to the NVME and i get proper sleep states. sleepstudy_nvme.html

So Windows 10 sleeps properly with NVME encryption.

Thanks for confirming it, then it does sound like there is something we must be missing on Linux. As an experiment on your system can you please try to comment out this block and see what happens? https://github.com/torvalds/linux/blob/7ffca2bb9d8bf6813db50364b1dd2c02f58fb65e/drivers/nvme/host/pci.c#L2922

Do you have a kernel that i can install on fedora 34? Otherwise it will take some time to build my first kernel. I am using 5.14.0rc7 from here https://fedoraproject.org/wiki/Kernel_Vanilla_Repositories

No, I don't have binaries, this was just an idea I'm suggesting. However having talked to the power architect, the SOC requirements don't change from OPAL to non-OPAL. I expect you're going to still have problems by making that change.

So I don't think this is the right direction. I think what you want to do is use something like this: https://github.com/ahroach/opal_suspend

Another alternative is to use one the forks of sedutil: https://github.com/fabiogermann/sedutil. There is an upstream attempt at getting sleep support added with this stale PR: https://github.com/Drive-Trust-Alliance/sedutil/pull/190 Perhaps you can try one of those options, or even work on fixing up / rebasing that PR.

@agd5f as I can tell this isn't an AMD specific problem, so I think we should close this issue.

It works fine on Intel Laptops...

It works fine on Intel Laptops...

As I mentioned before, the architecture for Intel laptops when it comes to s0i3 are different than the architecture in the AMD SOC's on the market today. Renoir and Cezanne both require that the NVME SSD goes into D3 in order to accomplish modern standby.

Well i just switched to an unencrypted drive and a UEFI password. By the way sedutil-cli does not recognize the NVME so the encryption is proprietary lenovo stuff.

Now i just encounter https://bugzilla.kernel.org/show_bug.cgi?id=212457 So i can suspend and resume but the Intel AX200 is grumpy.

This is the new dmesg, but i guess it is not AMDs fault...

dmesg_ax200.txt

Well i just switched to an unencrypted drive and a UEFI password. By the way sedutil-cli does not recognize the NVME so the encryption is proprietary lenovo stuff.

I thought you said it was a TCG OPAL drive? That's a standard. If it is a Lenovo proprietary encryption, you'll need to talk to Lenovo about getting the details used for encryption and decryption to extend the kernel ioctl for this encryption stuff.

This is the new dmesg, but i guess it is not AMDs fault...

You should attach your logs to that bug for the iwlwifi maintainer to look at your situation better.

Something to mention off hand I notice in your logs the card supports both D3hot and D3cold:

[ 0.327823] pci 0000:03:00.0: PME# supported from D0 D3hot D3cold You might try to set the sysfs attribute d3cold_allowed in /sys/bus/pci/devices? If that helps, maybe iwlwifi needs to set policy accordingly..

The CPU supports over 4 GB/S AES-XTS per core so there would be a power penalty that you have to accept if suspend and encryption is desired.

d3cold_allowed is already set

[julius@fedora ~]$ cat /sys/bus/pci/devices/0000:03:00.0/d3cold_allowed
1

I will wait until 5.14 becomes official in Fedora, maybe it is a pre-release bug or fixed with distribution specific patches.

closed

mentioned in issue #1737 (closed)

Apologies to bump this old issue - but can you do an experiment?

If you set iommu=pt on the kernel command line can encrypted nvme resume properly with the proprietary Lenovo stuff?

@superm1 thank you for your inquiry.

Yes, iommu=pt is a workaround

dmesg1.txt

dmesg2.txt

iommu.txt

reopened

fwts.txt

and here the fwts output

Okay - I think I finally understand what's going on. This is a Lenovo specific implementation in the BIOS that is having problems.

Can you please share the output from nvme id-ctrl so we can record what ssd details you have in this bug report?

For now I suggest to keep iommu=pt which will unfortunately effectively disable the iommu for all devices but will work around the root issue. Properly addressing it will require a firmware update from Lenovo.

There are even more similar reports for other machines here https://forums.lenovo.com/t5/Other-Linux-Discussions/Linux-S0ix-support-on-AMD-platforms/m-p/5098254?page=15#5603278

nvme.txt

added s0ix label

OK from the internal investigation at AMD - both #1910 (closed) and #1689 (closed) are the same root cause. The Lenovo firmware needs to be fixed to avoid it.

There are a few ways that Lenovo can address it, and that will be up to them which way they choose to do it. Their firmware team can discuss details with AMD if necessary. From an end user perspective the best workaround until Lenovo has issued a fix will remain to use iommu=pt on the kernel command line.

As such, I'm closing this issue in AMD's bug tracker.

closed

mentioned in issue #1910 (closed)

@superm1 i have seen your patch for thinkpads here https://git.kernel.org/pub/scm/linux/kernel/git/pdx86/platform-drivers-x86.git/commit/drivers/platform/x86/thinkpad_acpi.c?h=review-hans&id=a50dfa903391d542025a92c92606425fbccc8323 but it seems to be missing ideapads.

Correct; I don't believe it will help for encrypted NVME resume (your problem) on the Ideapads. You should keep using the iommu=pt workaround there until a firmware solution is come up with.

To make encrypted NVME work you'll need that SMI. BTW - if you want to avoid using encrypted NVME, you can probably port the same set of patches to ideapad-laptop. Feel free to CC me on any patches if you would like a review.

reopened

closed

mentioned in issue #2004 (closed)

mentioned in commit nouveau@455cd867

mentioned in issue #2682 (closed)