Skip to content
Snippets Groups Projects
New look: Off

Cezanne Suspend/Resume Failures, even with latest s0ix patches

    • Closed Issue created by Jorge d'Argence @dargence

    Brief summary of the problem:

    Suspend is unreliable as the laptop will fail to recover from a suspend/resume cycle. The failure rate is unpredictable. Sometimes things will work for multiple cycles, other times it fails on the first one.

    Hardware description:

    • CPU: 5900HS
    • GPU: Cezanne
    • System Memory: 2x8GB Micron DDR4 3200
    • Display(s): Built in 120 Hz 1080p, External Monoprice 35in Zero-G 100Hz Monitor
    • Type of Diplay Connection: HDMI

    System information:

    • Distro name and Version: Endeavour and Fedora 34
    • Kernel version: 5.13.2 and 5.14.rc-1
    • Custom kernel: 5.13.2 built with s0ix patches from 5.14, but also d3hot delay quirk Patched 5.13 kernel tree
    • AMD package version: 21.20 firmware

    How to reproduce the issue:

    Boot into system, suspend via any method (close lid, GNOME Suspend option, or systemctl suspend). Try to resume. With the 5.13 kernels, the system usually suspends right away (blinking power LED), but there can be times where the LED doesn't blink. One can usually wake the system up again from this state. When the resume fails, the system will fail to wake up. No tty is available, and one has to force the power off. Looking at logs after reboot, there is no clear indication of failure. The system will log that it entered s2idle state and no other logs will be seen. Enabling PM Debug doesn't add more useful details

    I have a Lenovo USB C dock that I use with this system. To make sure the issue isn't tied to external factors, I have tried these cycles with everything disconnected. The cycles fail the same with the dock connected/disconnected, or external monitor connected/disconnected.

    We have a community Discord for ROG users and most G14 and G15 users with Cezanne hardware have similar issues. We've been trying patches from the Renoir suspend thread on 5.12, 5.13 and 5.14 with limited success. The G14 (my model) seems to be the one having the most failures. G15 units appear to have more success but still fail eventually.

    Attached files:

    Journalctl output from a boot that had one successful resume, followed by a failed suspend. Log ends with 'entering s2idle' but the laptop had to be forced off. failedsuspend.txt

    Edited

    Designs

      Child items ...

      2. Activity

      • Jorge d'Argence changed the description

        changed the description

      • Jorge d'Argence
        Jorge d'Argence @dargence ·
        Author

        Had a crash on the first suspend attempt. Power LED was blinking, but on resume I got a flash of white on the internal LCD.

        Jul 17 08:31:36 G14-Arch NetworkManager[620]: <info>  [1626535896.1486] device (p2p-dev-wlan0): state change: disconnected -> unmanaged (reason 'sleeping', sys-iface-state: 'managed')
Jul 17 08:31:36 G14-Arch NetworkManager[620]: <info>  [1626535896.1497] manager: NetworkManager state is now ASLEEP
Jul 17 08:31:36 G14-Arch NetworkManager[620]: <info>  [1626535896.1499] device (wlan0): state change: activated -> deactivating (reason 'sleeping', sys-iface-state: 'managed')
Jul 17 08:31:36 G14-Arch dbus-daemon[616]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.5' (uid=0 pid=620 comm="/usr/bin/NetworkManager --no-daemon ")
Jul 17 08:31:36 G14-Arch systemd[1]: Starting Network Manager Script Dispatcher Service...
Jul 17 08:31:36 G14-Arch dbus-daemon[616]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Jul 17 08:31:36 G14-Arch systemd[1]: Started Network Manager Script Dispatcher Service.
Jul 17 08:31:36 G14-Arch audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jul 17 08:31:36 G14-Arch kernel: audit: type=1130 audit(1626535896.159:221): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jul 17 08:31:36 G14-Arch kernel: wlan0: deauthenticating from f6:92:bf:98:69:aa by local choice (Reason: 3=DEAUTH_LEAVING)
Jul 17 08:31:36 G14-Arch dnsmasq[793]: no servers found in /etc/resolv.conf, will retry
Jul 17 08:31:36 G14-Arch gnome-shell[1370]: Timelines with detached actors are not supported
Jul 17 08:31:36 G14-Arch gnome-shell[1370]: asusctl-gex: [INFO] disabling asusctl-gex version 3.7.1-r1
Jul 17 08:31:36 G14-Arch gnome-shell[1370]: asusctl-gex: [INFO] Stopping Profile client...
Jul 17 08:31:36 G14-Arch gnome-shell[1370]: asusctl-gex: [INFO] Stopping Profile DBus client...
Jul 17 08:31:36 G14-Arch gnome-shell[1370]: asusctl-gex: [INFO] Stopping GfxMode client...
Jul 17 08:31:36 G14-Arch gnome-shell[1370]: asusctl-gex: [INFO] Stopping GfxMode DBus client...
Jul 17 08:31:36 G14-Arch gnome-shell[1370]: _shell_embedded_window_unmap: assertion 'SHELL_IS_EMBEDDED_WINDOW (window)' failed
Jul 17 08:31:36 G14-Arch gnome-shell[1370]: _shell_embedded_window_unmap: assertion 'SHELL_IS_EMBEDDED_WINDOW (window)' failed
Jul 17 08:31:36 G14-Arch gnome-shell[1370]: Could not create transient scope for PID 7421: GDBus.Error:org.freedesktop.DBus.Error.UnixProcessIdUnknown: Process with ID 7421 does not exist.
Jul 17 08:31:36 G14-Arch gnome-shell[1370]: JS ERROR: Error getting systemd inhibitor: Gio.IOErrorEnum: GDBus.Error:org.freedesktop.login1.OperationInProgress: The operation inhibition has been requested for is already running
                                            _promisify/proto[asyncFunc]/</<@resource:///org/gnome/gjs/modules/core/overrides/Gio.js:418:45
                                            ### Promise created here: ###
                                            inhibit@resource:///org/gnome/shell/misc/loginManager.js:196:35
                                            _syncInhibitor@resource:///org/gnome/shell/ui/screenShield.js:211:32
Jul 17 08:31:36 G14-Arch wpa_supplicant[698]: wlan0: CTRL-EVENT-DISCONNECTED bssid=f6:92:bf:98:69:aa reason=3 locally_generated=1
Jul 17 08:31:36 G14-Arch NetworkManager[620]: <info>  [1626535896.4150] device (wlan0): supplicant interface state: completed -> disconnected
Jul 17 08:31:36 G14-Arch NetworkManager[620]: <info>  [1626535896.4151] device (wlan0): state change: deactivating -> disconnected (reason 'sleeping', sys-iface-state: 'managed')
Jul 17 08:31:36 G14-Arch wpa_supplicant[698]: wlan0: CTRL-EVENT-SIGNAL-CHANGE above=0 signal=0 noise=9999 txrate=0
Jul 17 08:31:36 G14-Arch avahi-daemon[613]: Leaving mDNS multicast group on interface wlan0.IPv6 with address fe80::7345:dff4:186:aa8b.
Jul 17 08:31:36 G14-Arch avahi-daemon[613]: Interface wlan0.IPv6 no longer relevant for mDNS.
Jul 17 08:31:36 G14-Arch NetworkManager[620]: <info>  [1626535896.4166] dhcp4 (wlan0): canceled DHCP transaction
Jul 17 08:31:36 G14-Arch NetworkManager[620]: <info>  [1626535896.4167] dhcp4 (wlan0): state changed bound -> terminated
Jul 17 08:31:36 G14-Arch avahi-daemon[613]: Interface wlan0.IPv4 no longer relevant for mDNS.
Jul 17 08:31:36 G14-Arch avahi-daemon[613]: Leaving mDNS multicast group on interface wlan0.IPv4 with address 192.168.1.21.
Jul 17 08:31:36 G14-Arch gnome-shell[1370]: An active wireless connection, in infrastructure mode, involves no access point?
Jul 17 08:31:36 G14-Arch NetworkManager[620]: <info>  [1626535896.4191] device (wlan0): set-hw-addr: set MAC address to DE:B9:34:81:74:82 (scanning)
Jul 17 08:31:36 G14-Arch avahi-daemon[613]: Joining mDNS multicast group on interface wlan0.IPv4 with address 192.168.1.21.
Jul 17 08:31:36 G14-Arch avahi-daemon[613]: New relevant interface wlan0.IPv4 for mDNS.
Jul 17 08:31:36 G14-Arch avahi-daemon[613]: Registering new address record for 192.168.1.21 on wlan0.IPv4.
Jul 17 08:31:36 G14-Arch avahi-daemon[613]: Leaving mDNS multicast group on interface wlan0.IPv4 with address 192.168.1.21.
Jul 17 08:31:36 G14-Arch avahi-daemon[613]: Interface wlan0.IPv4 no longer relevant for mDNS.
Jul 17 08:31:36 G14-Arch NetworkManager[620]: <info>  [1626535896.4265] device (wlan0): state change: disconnected -> unmanaged (reason 'sleeping', sys-iface-state: 'managed')
Jul 17 08:31:36 G14-Arch NetworkManager[620]: <info>  [1626535896.4286] device (wlan0): set-hw-addr: reset MAC address to E0:D4:64:26:63:3A (unmanage)
Jul 17 08:31:36 G14-Arch systemd[1]: Reached target Sleep.
Jul 17 08:31:36 G14-Arch systemd[1]: Starting NVIDIA system suspend actions...
Jul 17 08:31:36 G14-Arch suspend[7426]: nvidia-suspend.service
Jul 17 08:31:36 G14-Arch logger[7426]: <13>Jul 17 08:31:36 suspend: nvidia-suspend.service
Jul 17 08:31:36 G14-Arch systemd[1]: nvidia-suspend.service: Deactivated successfully.
Jul 17 08:31:36 G14-Arch systemd[1]: Finished NVIDIA system suspend actions.
Jul 17 08:31:36 G14-Arch audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=nvidia-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jul 17 08:31:36 G14-Arch audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=nvidia-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jul 17 08:31:36 G14-Arch systemd[1]: Starting System Suspend...
Jul 17 08:31:36 G14-Arch kernel: audit: type=1130 audit(1626535896.446:222): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=nvidia-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jul 17 08:31:36 G14-Arch kernel: audit: type=1131 audit(1626535896.449:223): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=nvidia-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jul 17 08:31:36 G14-Arch systemd-sleep[7428]: Entering sleep state 'suspend'...
Jul 17 08:31:36 G14-Arch kernel: PM: suspend entry (s2idle)
Jul 17 08:31:36 G14-Arch wpa_supplicant[698]: nl80211: deinit ifname=p2p-dev-wlan0 disabled_11b_rates=0
Jul 17 08:31:36 G14-Arch kernel: Filesystems sync: 0.059 seconds
Jul 17 08:31:36 G14-Arch kernel: PM: Preparing system for sleep (s2idle)
Jul 17 08:31:36 G14-Arch wpa_supplicant[698]: nl80211: deinit ifname=wlan0 disabled_11b_rates=0
        Edited by Jorge d'Argence
      • Mario Limonciello
        Mario Limonciello @superm1 ·
        Developer

        I notice a reference to a service nvidia-suspend. Do you have nvidia discrete graphics in this laptop as well?

        Are you using nvidia.ko or nouveau? A full dmesg may be useful to see more of the setup you have.

      • Jorge d'Argence
        Jorge d'Argence @dargence ·
        Author

        Yes, it has a discrete card and we are not using nouveau. I am using the nvidia 470 beta driver but I am also running in integrated mode at the moment so the nvidia GPU is not active.

        We found before that enabling the nvidia suspend/resume services helped with stability even on iGPU mode, so I've had it on since then. I'm happy to disable the services if we want to test that way.

        I also have a Lenovo USB C Docking station attached that seems to have a bad component, so ignore 'USB Cable is bad' messages.

        dmesg attachedG14dmesgOutput.txt

      • Mario Limonciello
        Mario Limonciello @superm1 ·
        Developer

        Have you configured the Nvidia driver for rtd3?

      • Jorge d'Argence
        Jorge d'Argence @dargence ·
        Author

        I haven't specifically done it. The nvidia driver is not even loaded when running in integrated mode, so the service won't do anything unless running in hybrid or dGPU mode. On this model of the ASUS ROG G14 (and the G15), the nvidia dGPU is completely off when in integrated mode and can be suspended completely in hybrid mode. It doesn't even show up in lspci in integrated mode:

        00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Renoir IOMMU
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge
00:02.4 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 51)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 166a
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 166b
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 166c
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 166d
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 166e
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 166f
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1670
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1671
02:00.0 Network controller: Intel Corporation Wi-Fi 6 AX200 (rev 1a)
03:00.0 Non-Volatile memory controller: SK hynix Device 174a
04:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cezanne (rev c4)
04:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device 1637
04:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor
04:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1
04:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1
04:00.5 Multimedia controller: Advanced Micro Devices, Inc. [AMD] Raven/Raven2/FireFlight/Renoir Audio Processor (rev 01)
04:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller
      • Mario Limonciello
        Mario Limonciello @superm1 ·
        Developer

        Where do you configure integrated mode? In BIOS setup? Is that what causes it to drop off PCI list or is it something else?

        The reason for my concern is that whatever way it's getting disabled may not be sufficient for the requirements for s0i3.

      • Jorge d'Argence
        Jorge d'Argence @dargence ·
        Author

        It is done through asusctl

        Graphics mode switching between iGPU, dGPU, on-demand, and vfio (for VM pass-through)

        I am not an expert on the methods used to set the GPU states. Luke Jones is the dev doing that work. I sent him a link to this comment so he can add any details that might be helpful.

        Edited by Jorge d'Argence
      • Mario Limonciello
        Mario Limonciello @superm1 ·
        Developer

        I would worry that the changes being made through such a tool turn off the dGPU but perhaps the firmware is still configured to expect it for the purposes of s0i3.

        If possible can you instead configure for hybrid mode and enable rtd3?

      • Jorge d'Argence
        Jorge d'Argence @dargence ·
        Author

        I'll try to break it in hybrid mode. rtd3 is enabled

        jorge@G14-Arch:~$ cat /proc/driver/nvidia/gpus/0000:01:00.0/power
Runtime D3 status:          Enabled (fine-grained)
Video Memory:               Off

GPU Hardware Support:
 Video Memory Self Refresh: Supported
 Video Memory Off:          Supported

Power Limits:
 Default:                   N/A milliwatts
 GPU Boost:                 N/A milliwatts
        Edited by Jorge d'Argence
      • Mario Limonciello
        Mario Limonciello @superm1 ·
        Developer

        Unless it's changed in new releases there's more to the configuration that you need to do. You should check NVIDIA's documentation on it. https://download.nvidia.com/XFree86/Linux-x86_64/435.17/README/dynamicpowermanagement.html

      • Jorge d'Argence
        Jorge d'Argence @dargence ·
        Author

        Thanks. As part of asusd/asusctl, we have

        options nvidia NVreg_DynamicPowerManagement=0x02 in modprobe and

        ACTION=="bind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030000", TEST=="power/control", ATTR{power/control}="auto"
ACTION=="bind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030200", TEST=="power/control", ATTR{power/control}="auto"

# Disable runtime PM for NVIDIA VGA/3D controller devices on driver unbind
ACTION=="unbind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030000", TEST=="power/control", ATTR{power/control}="on"
ACTION=="unbind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030200", TEST=="power/control", ATTR{power/control}="on"

        in a udev rule.

        We don't have the "Audio device PCI function, USB xHCI Host Controller function as well as USB Type-C UCSI Controller PCI function." removal lines but the workarounds are probably not needed for kernel versions 5.12+ since the document talks about kernel 4.19.

        I've been testing suspend/resume cycles in Hybrid mode all day today and have had 7 successful cycles. I usually have a crash within the first two cycles.

        I will continue collecting data and trying to make this crash and report back.

        Edited by Jorge d'Argence
      • Mario Limonciello
        Mario Limonciello @superm1 ·
        Developer

        Do you have any updates? If you continue to reproduce a crash I have a few next steps for you.

        1. Apply https://lists.freedesktop.org/archives/amd-gfx/2021-July/066768.html
        2. Apply:
        diff --git a/drivers/platform/x86/amd-pmc.c b/drivers/platform/x86/amd-pmc.c
index 680f94c7e075..ae67d1c999ee 100644
--- a/drivers/platform/x86/amd-pmc.c
+++ b/drivers/platform/x86/amd-pmc.c
@@ -269,6 +269,7 @@ static int amd_pmc_send_cmd(struct amd_pmc_dev *dev, bool set, u32 *data, u8 msg
        u32 val;

        mutex_lock(&dev->lock);
+       amd_pmc_dump_registers(dev);
        /* Wait until we get a valid response */
        rc = readx_poll_timeout(ioread32, dev->regbase + AMD_PMC_REGISTER_RESPONSE,
                                val, val != 0, PMC_MSG_DELAY_MIN_US,
        1. Turn on dynamic debugging for amd-pmc module.
        2. Share failing journal/dmesg.
      • Scott Bruce
        Scott Bruce @smbruce ·

        Quick update: we've been working around this issue with the delay hack from #1230 (closed) and #1629 (closed), in testing so far it seems like https://lkml.org/lkml/2021/9/13/2516 resolves it.

      • lijo lazar mentioned in issue #1708 (closed)

        mentioned in issue #1708 (closed)

        • Marco Laux
          Marco Laux @marcaux ·

          I'm not sure if this is of additional use but I also got a suspend failure with the latest delay patches:

          Sep 19 12:52:06 powerbookg14 systemd[1]: Starting System Suspend...
Sep 19 12:52:06 powerbookg14 systemd-sleep[9322]: Suspending system...
Sep 19 12:52:06 powerbookg14 kernel: PM: suspend entry (s2idle)
Sep 19 12:52:07 powerbookg14 kernel: Filesystems sync: 0.031 seconds
Sep 19 12:52:13 powerbookg14 kernel: Freezing user space processes ... (elapsed 0.005 seconds) done.
Sep 19 12:52:13 powerbookg14 kernel: OOM killer disabled.
Sep 19 12:52:13 powerbookg14 kernel: Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
Sep 19 12:52:13 powerbookg14 kernel: printk: Suspending console(s) (use no_console_suspend to debug)
Sep 19 12:52:13 powerbookg14 kernel: PM: suspend devices took 0.280 seconds
Sep 19 12:52:13 powerbookg14 kernel: amd_pmc AMDI0005:00: SMU response timed out
Sep 19 12:52:13 powerbookg14 kernel: amd_pmc AMDI0005:00: SMU response timed out
Sep 19 12:52:13 powerbookg14 kernel: amd_pmc AMDI0005:00: suspend failed
Sep 19 12:52:13 powerbookg14 kernel: PM: dpm_run_callback(): acpi_subsys_suspend_noirq+0x0/0x40 returns -110
Sep 19 12:52:13 powerbookg14 kernel: amd_pmc AMDI0005:00: PM: failed to suspend noirq: error -110
Sep 19 12:52:13 powerbookg14 kernel: PM: noirq suspend of devices failed
Sep 19 12:52:13 powerbookg14 kernel: pci 0000:00:00.2: can't derive routing for PCI INT A
Sep 19 12:52:13 powerbookg14 kernel: pci 0000:00:00.2: PCI INT A: no GSI
Sep 19 12:52:13 powerbookg14 kernel: [drm] PCIE GART of 1024M enabled.
Sep 19 12:52:13 powerbookg14 kernel: [drm] PTB located at 0x000000F400900000
Sep 19 12:52:13 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: SMU is resuming...
Sep 19 12:52:13 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: smu fw reported version = 0x00402c00 (64.44.0)
Sep 19 12:52:13 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: dpm has been disabled
Sep 19 12:52:13 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: SMU is resumed successfully!
Sep 19 12:52:13 powerbookg14 kernel: amdgpu 0000:04:00.0: [drm:amdgpu_ring_test_helper [amdgpu]] *ERROR* ring sdma0 test failed (-110)
Sep 19 12:52:13 powerbookg14 kernel: [drm:amdgpu_device_ip_resume_phase2 [amdgpu]] *ERROR* resume of IP block <sdma_v4_0> failed -110
Sep 19 12:52:13 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: amdgpu_device_ip_resume failed (-110).
Sep 19 12:52:13 powerbookg14 kernel: PM: dpm_run_callback(): pci_pm_resume+0x0/0xe0 returns -110
Sep 19 12:52:13 powerbookg14 kernel: amdgpu 0000:04:00.0: PM: failed to resume async: error -110
[...]
Sep 19 12:52:13 powerbookg14 kernel: PM: resume devices took 0.281 seconds
[...]
Sep 19 12:52:13 powerbookg14 kernel: Restarting tasks ... done.
[...]
Sep 19 12:52:13 powerbookg14 kernel: PM: suspend exit
Sep 19 12:52:13 powerbookg14 kernel: PM: suspend entry (s2idle)
Sep 19 12:52:13 powerbookg14 kernel: Filesystems sync: 0.006 seconds
Sep 19 12:52:16 powerbookg14 kernel: Freezing user space processes ... (elapsed 0.003 seconds) done.
Sep 19 12:52:16 powerbookg14 kernel: OOM killer disabled.
Sep 19 12:52:16 powerbookg14 kernel: Freezing remaining freezable tasks ... (elapsed 0.655 seconds) done.
Sep 19 12:52:16 powerbookg14 kernel: printk: Suspending console(s) (use no_console_suspend to debug)
Sep 19 12:52:16 powerbookg14 kernel: ------------[ cut here ]------------
Sep 19 12:52:16 powerbookg14 kernel: WARNING: CPU: 2 PID: 9348 at drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm.c:2008 dm_suspend+0x186/0x1a0 [amdgpu]
Sep 19 12:52:16 powerbookg14 kernel: Modules linked in: uinput rfcomm snd_seq_dummy snd_hrtimer xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_nat_tftp nf_conntrack_tftp bridge stp llc nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_>
Sep 19 12:52:16 powerbookg14 kernel:  rapl cfg80211 snd_timer snd_rn_pci_acp3x pcspkr ucsi_acpi joydev snd asus_nb_wmi wmi_bmof typec_ucsi ecdh_generic snd_pci_acp3x soundcore i2c_piix4 typec asus_wireless amd_pmc acpi_cpufreq v4l2loopback(OE) videodev >
Sep 19 12:52:16 powerbookg14 kernel: CPU: 2 PID: 9348 Comm: kworker/u32:29 Tainted: P           OE     5.14.3-300.rog.fc34.x86_64 #1
Sep 19 12:52:16 powerbookg14 kernel: Hardware name: ASUSTeK COMPUTER INC. ROG Zephyrus G14 GA401QM/GA401QM, BIOS GA401QM.408 05/13/2021
Sep 19 12:52:16 powerbookg14 kernel: Workqueue: events_unbound async_run_entry_fn
Sep 19 12:52:16 powerbookg14 kernel: RIP: 0010:dm_suspend+0x186/0x1a0 [amdgpu]
Sep 19 12:52:16 powerbookg14 kernel: Code: c3 31 d2 4c 89 e6 4c 89 ef e8 b6 30 15 00 83 f8 01 74 1e 89 c2 48 c7 c6 e0 09 a3 c0 48 c7 c7 28 bd af c0 e8 cc 23 d9 ff eb b8 <0f> 0b e9 9f fe ff ff 4c 89 e6 4c 89 ef e8 a8 84 14 00 eb a4 66 0f
Sep 19 12:52:16 powerbookg14 kernel: RSP: 0018:ffffbc47c7aebd00 EFLAGS: 00010282
Sep 19 12:52:16 powerbookg14 kernel: RAX: 0000000000000000 RBX: ffff95c0ce855610 RCX: 0000000000000000
Sep 19 12:52:16 powerbookg14 kernel: RDX: 000000000000000a RSI: 00000000000161ba RDI: ffff95c0ce840000
Sep 19 12:52:16 powerbookg14 kernel: RBP: ffff95c0ce840000 R08: 0000000000000000 R09: 00000000888881ff
Sep 19 12:52:16 powerbookg14 kernel: R10: 0000000000000000 R11: 0000000000000002 R12: ffff95c0ce840000
Sep 19 12:52:16 powerbookg14 kernel: R13: 0000000000000000 R14: 0000000000000002 R15: 0000000000000000
Sep 19 12:52:16 powerbookg14 kernel: FS:  0000000000000000(0000) GS:ffff95cb7e680000(0000) knlGS:0000000000000000
Sep 19 12:52:16 powerbookg14 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 19 12:52:16 powerbookg14 kernel: CR2: 00000ef7612c7240 CR3: 000000093cc10000 CR4: 0000000000750ee0
Sep 19 12:52:16 powerbookg14 kernel: PKRU: 55555554
Sep 19 12:52:16 powerbookg14 kernel: Call Trace:
Sep 19 12:52:16 powerbookg14 kernel:  ? soc15_common_set_clockgating_state+0x16c/0x180 [amdgpu]
Sep 19 12:52:16 powerbookg14 kernel:  ? amdgpu_device_set_cg_state+0x92/0xe0 [amdgpu]
Sep 19 12:52:16 powerbookg14 kernel:  amdgpu_device_ip_suspend_phase1+0x63/0xc0 [amdgpu]
Sep 19 12:52:16 powerbookg14 kernel:  amdgpu_device_suspend+0x62/0xc0 [amdgpu]
Sep 19 12:52:16 powerbookg14 kernel:  amdgpu_pmops_suspend+0x32/0x40 [amdgpu]
Sep 19 12:52:16 powerbookg14 kernel:  pci_pm_suspend+0x74/0x160
Sep 19 12:52:16 powerbookg14 kernel:  ? pci_pm_freeze+0xb0/0xb0
Sep 19 12:52:16 powerbookg14 kernel:  dpm_run_callback+0x49/0x120
Sep 19 12:52:16 powerbookg14 kernel:  __device_suspend+0x11c/0x4c0
Sep 19 12:52:16 powerbookg14 kernel:  async_suspend+0x1b/0x90
Sep 19 12:52:16 powerbookg14 kernel:  async_run_entry_fn+0x30/0x130
Sep 19 12:52:16 powerbookg14 kernel:  process_one_work+0x1ec/0x390
Sep 19 12:52:16 powerbookg14 kernel:  worker_thread+0x53/0x3e0
Sep 19 12:52:16 powerbookg14 kernel:  ? process_one_work+0x390/0x390
Sep 19 12:52:16 powerbookg14 kernel:  kthread+0x127/0x150
Sep 19 12:52:16 powerbookg14 kernel:  ? set_kthread_struct+0x40/0x40
Sep 19 12:52:16 powerbookg14 kernel:  ret_from_fork+0x22/0x30
Sep 19 12:52:16 powerbookg14 kernel: ---[ end trace 6256c620ca761a3a ]---
Sep 19 12:52:16 powerbookg14 kernel: PM: suspend devices took 0.175 seconds
Sep 19 12:52:16 powerbookg14 kernel: amd_pmc AMDI0005:00: SMU response timed out
Sep 19 12:52:16 powerbookg14 kernel: amd_pmc AMDI0005:00: suspend failed
Sep 19 12:52:16 powerbookg14 kernel: PM: dpm_run_callback(): acpi_subsys_suspend_noirq+0x0/0x40 returns -110
Sep 19 12:52:16 powerbookg14 kernel: amd_pmc AMDI0005:00: PM: failed to suspend noirq: error -110
Sep 19 12:52:16 powerbookg14 kernel: PM: noirq suspend of devices failed
Sep 19 12:52:16 powerbookg14 kernel: pci 0000:00:00.2: can't derive routing for PCI INT A
Sep 19 12:52:16 powerbookg14 kernel: pci 0000:00:00.2: PCI INT A: no GSI
Sep 19 12:52:16 powerbookg14 kernel: [drm] PCIE GART of 1024M enabled.
Sep 19 12:52:16 powerbookg14 kernel: [drm] PTB located at 0x000000F400900000
Sep 19 12:52:16 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: SMU is resuming...
Sep 19 12:52:16 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: smu fw reported version = 0x00402c00 (64.44.0)
Sep 19 12:52:16 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: dpm has been disabled
Sep 19 12:52:16 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: SMU is resumed successfully!
Sep 19 12:52:16 powerbookg14 kernel: nvme nvme0: Shutdown timeout set to 8 seconds
Sep 19 12:52:16 powerbookg14 kernel: nvme nvme0: 16/0/0 default/read/poll queues
Sep 19 12:52:16 powerbookg14 kernel: amdgpu 0000:04:00.0: [drm:amdgpu_ring_test_helper [amdgpu]] *ERROR* ring sdma0 test failed (-110)
Sep 19 12:52:16 powerbookg14 kernel: [drm:amdgpu_device_ip_resume_phase2 [amdgpu]] *ERROR* resume of IP block <sdma_v4_0> failed -110
Sep 19 12:52:16 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: amdgpu_device_ip_resume failed (-110).
Sep 19 12:52:16 powerbookg14 kernel: PM: dpm_run_callback(): pci_pm_resume+0x0/0xe0 returns -110
Sep 19 12:52:16 powerbookg14 kernel: amdgpu 0000:04:00.0: PM: failed to resume async: error -110
Sep 19 12:52:16 powerbookg14 kernel: PM: resume devices took 0.233 seconds
Sep 19 12:52:16 powerbookg14 kernel: OOM killer enabled.
Sep 19 12:52:16 powerbookg14 kernel: Restarting tasks ... done.
Sep 19 12:52:16 powerbookg14 systemd-sleep[9322]: Failed to suspend system. System resumed again: Connection timed out
Sep 19 12:52:16 powerbookg14 kernel: PM: suspend exit
[...]
Sep 19 12:52:16 powerbookg14 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 19 12:52:16 powerbookg14 systemd[1]: systemd-suspend.service: Main process exited, code=exited, status=1/FAILURE
Sep 19 12:52:16 powerbookg14 systemd[1]: systemd-suspend.service: Failed with result 'exit-code'.
Sep 19 12:52:16 powerbookg14 systemd[1]: Failed to start System Suspend.
Sep 19 12:52:16 powerbookg14 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Sep 19 12:52:16 powerbookg14 systemd[1]: Dependency failed for Suspend.
Sep 19 12:52:16 powerbookg14 systemd[1]: suspend.target: Job suspend.target/start failed with result 'dependency'.
Sep 19 12:52:16 powerbookg14 systemd-logind[1568]: Operation 'sleep' finished.
Sep 19 12:52:16 powerbookg14 systemd[1]: Stopped target Sleep.
[...]
Sep 19 12:52:16 powerbookg14 systemd[1]: Starting NVIDIA system resume actions...
Sep 19 12:52:16 powerbookg14 suspend[9366]: nvidia-resume.service
Sep 19 12:52:16 powerbookg14 logger[9366]: <13>Sep 19 12:52:16 suspend: nvidia-resume.service
[...]
Sep 19 12:52:17 powerbookg14 systemd[1]: nvidia-resume.service: Deactivated successfully.
Sep 19 12:52:17 powerbookg14 systemd[1]: Finished NVIDIA system resume actions.
Sep 19 12:52:17 powerbookg14 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=nvidia-resume comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 19 12:52:17 powerbookg14 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=nvidia-resume comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 19 12:52:17 powerbookg14 kernel: rfkill: input handler disabled
[...]
Sep 19 12:52:17 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: couldn't schedule ib on ring <sdma0>
Sep 19 12:52:17 powerbookg14 kernel: [drm:amdgpu_job_run [amdgpu]] *ERROR* Error scheduling IBs (-22)
[...]
Sep 19 12:52:17 powerbookg14 kernel: gmc_v9_0_process_interrupt: 110 callbacks suppressed
Sep 19 12:52:17 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: [gfxhub0] retry page fault (src_id:0 ring:0 vmid:3 pasid:32769, for process gnome-shell pid 2728 thread gnome-shel:cs0 pid 2766)
Sep 19 12:52:17 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu:   in page starting at address 0x000080011a910000 from IH client 0x1b (UTCL2)
Sep 19 12:52:17 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: VM_L2_PROTECTION_FAULT_STATUS:0x00341051
Sep 19 12:52:17 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu:          Faulty UTCL2 client ID: TCP (0x8)
Sep 19 12:52:17 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu:          MORE_FAULTS: 0x1
Sep 19 12:52:17 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu:          WALKER_ERROR: 0x0
Sep 19 12:52:17 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu:          PERMISSION_FAULTS: 0x5
Sep 19 12:52:17 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu:          MAPPING_ERROR: 0x0
Sep 19 12:52:17 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu:          RW: 0x1
Sep 19 12:52:17 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: couldn't schedule ib on ring <sdma0>
Sep 19 12:52:17 powerbookg14 kernel: [drm:amdgpu_job_run [amdgpu]] *ERROR* Error scheduling IBs (-22)
Sep 19 12:52:17 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: [gfxhub0] retry page fault (src_id:0 ring:0 vmid:3 pasid:32769, for process gnome-shell pid 2728 thread gnome-shel:cs0 pid 2766)
[...]
Sep 19 12:52:17 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: couldn't schedule ib on ring <sdma0>
Sep 19 12:52:17 powerbookg14 kernel: [drm:amdgpu_job_run [amdgpu]] *ERROR* Error scheduling IBs (-22)
[...]

          full log here: journal.txt

          Then it endlessly loops through

          kernel: amdgpu 0000:04:00.0: amdgpu: couldn't schedule ib on ring <sdma0>
kernel: [drm:amdgpu_job_run [amdgpu]] *ERROR* Error scheduling IBs (-22)
        • Mario Limonciello
          Mario Limonciello @superm1 ·
          Developer

          Would you mind also applying the idle mask patches so we can see what your idle mask is both with and without this failure?

          After applying set amd_pmc.dyndbg="+p" on your kernel command line so it turns on the extra debugging output.

          Edited by Mario Limonciello
        • Marco Laux
          Marco Laux @marcaux ·

          On the previous boot I sadly did not have set amd_pmc.dyndbg="+p".

          Now on another boot cycle I have it set (the custom kernel 5.14.3-300.rog.fc34.x86_64 contains all the patches as far as I know) and another try to hibernate failed.

          This is what I got from journalctl:

          Command line: BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.14.3-300.rog.fc34.x86_64 [...] rhgb quiet rd.driver.blacklist=nouveau modprobe.blacklist=nouveau nvidia-drm.modeset=0 pm_debug_messages amd_pmc.dyndbg=+p "acpi.dyndbg=file drivers/acpi/x86/s2idle.c +p"
[...]
Sep 19 19:40:21 powerbookg14 systemd[1]: Reached target Sleep.
Sep 19 19:40:22 powerbookg14 systemd[1]: Starting NVIDIA system hibernate actions...
Sep 19 19:40:22 powerbookg14 hibernate[22451]: nvidia-hibernate.service
Sep 19 19:40:22 powerbookg14 logger[22451]: <13>Sep 19 19:40:22 hibernate: nvidia-hibernate.service
Sep 19 19:40:22 powerbookg14 kernel: rfkill: input handler enabled
[...]
Sep 19 19:40:22 powerbookg14 uresourced[1889]: Setting resources on user-1000.slice (MemoryMin: 0, MemoryLow: 0, CPUWeight: 100, IOWeight: 100)
Sep 19 19:40:22 powerbookg14 uresourced[1889]: Setting resources on user@1000.service (MemoryMin: 0, MemoryLow: 0, CPUWeight: 100, IOWeight: 100)
Sep 19 19:40:22 powerbookg14 uresourced[1889]: Setting resources on user.slice (MemoryMin: 0, MemoryLow: 0, CPUWeight: -, IOWeight: -)
Sep 19 19:40:23 powerbookg14 systemd[1]: nvidia-hibernate.service: Deactivated successfully.
Sep 19 19:40:23 powerbookg14 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=nvidia-hibernate comm="systemd" exe="/usr/lib/systemd>
Sep 19 19:40:23 powerbookg14 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=nvidia-hibernate comm="systemd" exe="/usr/lib/systemd/>
Sep 19 19:40:23 powerbookg14 systemd[1]: Finished NVIDIA system hibernate actions.
Sep 19 19:40:23 powerbookg14 systemd[1]: nvidia-hibernate.service: Consumed 1.278s CPU time.
Sep 19 19:40:23 powerbookg14 systemd[1]: Starting Hibernate...
Sep 19 19:40:23 powerbookg14 kernel: PM: Hibernation mode set to 'platform'
Sep 19 19:40:23 powerbookg14 systemd-sleep[22511]: Suspending system...
Sep 19 19:40:23 powerbookg14 kernel: PM: hibernation: hibernation entry
Sep 19 19:40:31 powerbookg14 kernel: Filesystems sync: 0.042 seconds
Sep 19 19:40:31 powerbookg14 kernel: Freezing user space processes ... (elapsed 0.005 seconds) done.
Sep 19 19:40:31 powerbookg14 kernel: OOM killer disabled.
Sep 19 19:40:31 powerbookg14 kernel: PM: hibernation: Marking nosave pages: [mem 0x00000000-0x00000fff]
Sep 19 19:40:31 powerbookg14 kernel: PM: hibernation: Marking nosave pages: [mem 0x000a0000-0x000fffff]
Sep 19 19:40:31 powerbookg14 kernel: PM: hibernation: Marking nosave pages: [mem 0x09dbf000-0x0a000fff]
Sep 19 19:40:31 powerbookg14 kernel: PM: hibernation: Marking nosave pages: [mem 0x0a200000-0x0a20efff]
Sep 19 19:40:31 powerbookg14 kernel: PM: hibernation: Marking nosave pages: [mem 0xdaf22000-0xdaf22fff]
Sep 19 19:40:31 powerbookg14 kernel: PM: hibernation: Marking nosave pages: [mem 0xe6fec000-0xe6fecfff]
Sep 19 19:40:31 powerbookg14 kernel: PM: hibernation: Marking nosave pages: [mem 0xe6ff9000-0xe6ff9fff]
Sep 19 19:40:31 powerbookg14 kernel: PM: hibernation: Marking nosave pages: [mem 0xe7005000-0xe7005fff]
Sep 19 19:40:31 powerbookg14 kernel: PM: hibernation: Marking nosave pages: [mem 0xe7692000-0xe773afff]
Sep 19 19:40:31 powerbookg14 kernel: PM: hibernation: Marking nosave pages: [mem 0xe9478000-0xe9478fff]
Sep 19 19:40:31 powerbookg14 kernel: PM: hibernation: Marking nosave pages: [mem 0xeae0a000-0xecdfefff]
Sep 19 19:40:31 powerbookg14 kernel: PM: hibernation: Marking nosave pages: [mem 0xee000000-0xffffffff]
Sep 19 19:40:31 powerbookg14 kernel: PM: hibernation: Basic memory bitmaps created
Sep 19 19:40:31 powerbookg14 kernel: PM: hibernation: Preallocating image memory
Sep 19 19:40:31 powerbookg14 kernel: PM: hibernation: Allocated 3741700 pages for snapshot
Sep 19 19:40:31 powerbookg14 kernel: PM: hibernation: Allocated 14966800 kbytes in 0.52 seconds (28782.30 MB/s)
Sep 19 19:40:31 powerbookg14 kernel: Freezing remaining freezable tasks ... (elapsed 0.002 seconds) done.
Sep 19 19:40:31 powerbookg14 kernel: printk: Suspending console(s) (use no_console_suspend to debug)
Sep 19 19:40:31 powerbookg14 kernel: [drm] free PSP TMR buffer
Sep 19 19:40:31 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: MODE2 reset
Sep 19 19:40:31 powerbookg14 kernel: PM: freeze of devices complete after 263.643 msecs
Sep 19 19:40:31 powerbookg14 kernel: PM: late freeze of devices complete after 1.104 msecs
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:1
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:0
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:3
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: SMU response timed out
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:0
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:0
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:7
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:0
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:0
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:7
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: failed to talk to SMU
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:0
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:0
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:7
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:0
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:0
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:7
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: failed to talk to SMU
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:0
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:0
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:7
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: suspend failed
Sep 19 19:40:31 powerbookg14 kernel: PM: dpm_run_callback(): amd_pmc_suspend+0x0/0x80 [amd_pmc] returns -110
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: PM: failed to freeze noirq: error -110
Sep 19 19:40:31 powerbookg14 kernel: PM: noirq freeze of devices aborted after 6000.797 msecs
Sep 19 19:40:31 powerbookg14 kernel: PM: noirq recover of devices complete after 0.219 msecs
Sep 19 19:40:31 powerbookg14 kernel: PM: early recover of devices complete after 0.382 msecs
Sep 19 19:40:31 powerbookg14 kernel: PM: end freeze of devices aborted after 6002.659 msecs
Sep 19 19:40:31 powerbookg14 kernel: PM: hibernation: Some devices failed to power down, aborting
Sep 19 19:40:31 powerbookg14 kernel: pci 0000:00:00.2: can't derive routing for PCI INT A
Sep 19 19:40:31 powerbookg14 kernel: pci 0000:00:00.2: PCI INT A: no GSI
Sep 19 19:40:31 powerbookg14 kernel: usb usb1: root hub lost power or was reset
Sep 19 19:40:31 powerbookg14 kernel: usb usb2: root hub lost power or was reset
Sep 19 19:40:31 powerbookg14 kernel: usb usb3: root hub lost power or was reset
Sep 19 19:40:31 powerbookg14 kernel: usb usb4: root hub lost power or was reset
Sep 19 19:40:31 powerbookg14 kernel: [drm] PCIE GART of 1024M enabled.
Sep 19 19:40:31 powerbookg14 kernel: [drm] PTB located at 0x000000F400900000
Sep 19 19:40:31 powerbookg14 kernel: [drm] PSP is resuming...
Sep 19 19:40:31 powerbookg14 kernel: [drm] reserve 0x400000 from 0xf41f800000 for PSP TMR
Sep 19 19:40:31 powerbookg14 kernel: nvme nvme0: Shutdown timeout set to 8 seconds
Sep 19 19:40:31 powerbookg14 kernel: nvme nvme0: 16/0/0 default/read/poll queues
Sep 19 19:40:31 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: RAS: optional ras ta ucode is not available
Sep 19 19:40:31 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: RAP: optional rap ta ucode is not available
Sep 19 19:40:31 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: SECUREDISPLAY: securedisplay ta ucode is not available
Sep 19 19:40:31 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: SMU is resuming...
Sep 19 19:40:31 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: smu fw reported version = 0x00402c00 (64.44.0)
Sep 19 19:40:31 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: dpm has been disabled
Sep 19 19:40:31 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: SMU is resumed successfully!
Sep 19 19:40:31 powerbookg14 kernel: usb 3-3: reset full-speed USB device number 2 using xhci_hcd
Sep 19 19:40:31 powerbookg14 kernel: usb 1-4: reset full-speed USB device number 3 using xhci_hcd
Sep 19 19:40:31 powerbookg14 kernel: [drm] kiq ring mec 2 pipe 1 q 0
Sep 19 19:40:31 powerbookg14 kernel: [drm] DMUB hardware initialized: version=0x01010019
Sep 19 19:40:31 powerbookg14 kernel: usb 3-4: reset full-speed USB device number 3 using xhci_hcd
Sep 19 19:40:31 powerbookg14 kernel: usb 1-3: reset full-speed USB device number 2 using xhci_hcd
Sep 19 19:40:31 powerbookg14 kernel: [drm] VCN decode and encode initialized successfully(under DPG Mode).
Sep 19 19:40:31 powerbookg14 kernel: [drm] JPEG decode initialized successfully.
Sep 19 19:40:31 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: ring gfx uses VM inv eng 0 on hub 0
Sep 19 19:40:31 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 1 on hub 0
Sep 19 19:40:31 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 4 on hub 0
Sep 19 19:40:31 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 5 on hub 0
Sep 19 19:40:31 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 6 on hub 0
Sep 19 19:40:31 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 7 on hub 0
Sep 19 19:40:31 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 8 on hub 0
Sep 19 19:40:31 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 9 on hub 0
Sep 19 19:40:31 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 10 on hub 0
Sep 19 19:40:31 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: ring kiq_2.1.0 uses VM inv eng 11 on hub 0
Sep 19 19:40:31 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: ring sdma0 uses VM inv eng 0 on hub 1
Sep 19 19:40:31 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: ring vcn_dec uses VM inv eng 1 on hub 1
Sep 19 19:40:31 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: ring vcn_enc0 uses VM inv eng 4 on hub 1
Sep 19 19:40:31 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: ring vcn_enc1 uses VM inv eng 5 on hub 1
Sep 19 19:40:31 powerbookg14 kernel: amdgpu 0000:04:00.0: amdgpu: ring jpeg_dec uses VM inv eng 6 on hub 1
Sep 19 19:40:31 powerbookg14 kernel: PM: restore of devices complete after 686.769 msecs
Sep 19 19:40:31 powerbookg14 kernel: PM: hibernation: Basic memory bitmaps freed
Sep 19 19:40:31 powerbookg14 kernel: OOM killer enabled.
Sep 19 19:40:31 powerbookg14 kernel: Restarting tasks ... done.
Sep 19 19:40:31 powerbookg14 kernel: hid-generic 0003:0B05:193B.0006: hiddev97,hidraw2: USB HID v1.10 Device [ITE Tech. Inc. ITE Device(8910)] on usb-0000:04:00.3-4/input0
Sep 19 19:40:31 powerbookg14 kernel: PM: hibernation: hibernation exit
Sep 19 19:40:31 powerbookg14 kernel: Bluetooth: hci0: Firmware revision 0.0 build 191 week 21 2021
Sep 19 19:40:31 powerbookg14 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-hibernate comm="systemd" exe="/usr/lib/system>
Sep 19 19:40:31 powerbookg14 systemd-sleep[22511]: Failed to suspend system. System resumed again: Connection timed out
Sep 19 19:40:31 powerbookg14 ModemManager[1511]: <info>  [sleep-monitor] system is resuming
Sep 19 19:40:31 powerbookg14 systemd[1]: Starting Load/Save RF Kill Switch Status...
Sep 19 19:40:31 powerbookg14 systemd[1]: systemd-hibernate.service: Main process exited, code=exited, status=1/FAILURE
Sep 19 19:40:31 powerbookg14 systemd[1]: systemd-hibernate.service: Failed with result 'exit-code'.
Sep 19 19:40:31 powerbookg14 systemd[1]: Failed to start Hibernate.
Sep 19 19:40:31 powerbookg14 systemd[1]: Dependency failed for System Hibernation.
Sep 19 19:40:31 powerbookg14 systemd[1]: hibernate.target: Job hibernate.target/start failed with result 'dependency'.
Sep 19 19:40:31 powerbookg14 upowerd[1374]: treating change event as add on /sys/devices/pci0000:00/0000:00:08.1/0000:04:00.3/usb1/1-4
Sep 19 19:40:31 powerbookg14 systemd[1]: systemd-hibernate.service: Consumed 1.281s CPU time.
Sep 19 19:40:31 powerbookg14 upowerd[1374]: treating change event as add on /sys/devices/pci0000:00/0000:00:08.1/0000:04:00.4/usb3/3-3
Sep 19 19:40:31 powerbookg14 systemd-logind[1537]: Operation 'sleep' finished.
Sep 19 19:40:31 powerbookg14 systemd[1]: Stopped target Sleep.
          Edited by Marco Laux
        • Marco Laux
          Marco Laux @marcaux ·

          This is the whole list of amd_pmc entries from the previous boot - I hope this is useful:

          $ journalctl -b-1 | grep amd_pmc

Sep 19 14:23:27 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:1
Sep 19 14:23:27 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:0
Sep 19 14:23:27 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:0
Sep 19 14:23:27 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:1
Sep 19 14:23:27 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:73f
Sep 19 14:23:27 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:9
Sep 19 14:23:27 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:1
Sep 19 14:23:27 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:73f
Sep 19 14:23:27 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:9
Sep 19 14:23:27 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:1
Sep 19 14:23:27 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:ec2fda80
Sep 19 14:23:27 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:5
Sep 19 14:23:27 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:1
Sep 19 14:23:27 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:ec2fda80
Sep 19 14:23:27 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:5
Sep 19 14:23:27 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:1
Sep 19 14:23:27 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:0
Sep 19 14:23:27 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:4
Sep 19 14:23:27 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:1
Sep 19 14:23:27 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:0
Sep 19 14:23:27 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:4
Sep 19 14:23:27 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:1
Sep 19 14:23:27 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:0
Sep 19 14:23:27 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:6
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:1
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:0
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:6
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:1
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:0
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:7
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:1
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:0
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:7
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:1
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:0
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:6
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:1
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:0
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:6
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:1
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:1
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:3
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:1
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:0
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:0
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:1
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:0
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:8
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:1
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:0
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:8
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:1
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:0
Sep 19 19:03:45 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:3
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:1
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:0
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:3
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: SMU response timed out
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:0
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:0
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:7
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:0
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:0
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:7
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: failed to talk to SMU
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:0
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:0
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:7
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:0
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:0
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:7
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: failed to talk to SMU
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_RESPONSE:0
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_ARGUMENT:0
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: AMD_PMC_REGISTER_MESSAGE:7
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: suspend failed
Sep 19 19:40:31 powerbookg14 kernel: PM: dpm_run_callback(): amd_pmc_suspend+0x0/0x80 [amd_pmc] returns -110
Sep 19 19:40:31 powerbookg14 kernel: amd_pmc AMDI0005:00: PM: failed to freeze noirq: error -110
          $ sudo cat /sys/kernel/debug/dri/0/amdgpu_firmware_info

VCE feature version: 0, firmware version: 0x00000000
UVD feature version: 0, firmware version: 0x00000000
MC feature version: 0, firmware version: 0x00000000
ME feature version: 52, firmware version: 0x000000a4
PFP feature version: 52, firmware version: 0x000000bc
CE feature version: 52, firmware version: 0x0000004f
RLC feature version: 1, firmware version: 0x0000003c
RLC SRLC feature version: 1, firmware version: 0x00000001
RLC SRLG feature version: 1, firmware version: 0x00000001
RLC SRLS feature version: 1, firmware version: 0x00000001
MEC feature version: 52, firmware version: 0x000001ca
SOS feature version: 0, firmware version: 0x00000000
ASD feature version: 0, firmware version: 0x2100005a
TA XGMI feature version: 0x00000000, firmware version: 0x00000000
TA RAS feature version: 0x00000000, firmware version: 0x00000000
TA HDCP feature version: 0x1700001f, firmware version: 0x00000000
TA DTM feature version: 0x12000009, firmware version: 0x00000000
TA RAP feature version: 0x00000000, firmware version: 0x00000000
TA SECUREDISPLAY feature version: 0x00000000, firmware version: 0x00000000
SMC feature version: 0, firmware version: 0x00402c00
SDMA0 feature version: 41, firmware version: 0x00000028
VCN feature version: 0, firmware version: 0x0510e014
DMCU feature version: 0, firmware version: 0x00000000
DMCUB feature version: 0, firmware version: 0x01010019
TOC feature version: 0, firmware version: 0x00000000
VBIOS version: 113-CEZANNE-018
          Edited by Marco Laux
        • Marco Laux
          Marco Laux @marcaux ·

          ah if you mean this patchset: https://patchwork.kernel.org/project/platform-driver-x86/patch/20210916124002.2529-1-Sanket.Goswami@amd.com/ I think this is not yet in there. I'll have a look to apply it asap. Will this probably also be in 5.14.6? This should soon also be in Fedora for me then.

          Edited by Marco Laux
        • Mario Limonciello
          Mario Limonciello @superm1 ·
          Developer

          Yeah that's the patchset i'm referring to. Unfortunately I don't think it is likely to land until 5.16. So I would suggest you backport it manually for now.

        • Mario Limonciello
          Mario Limonciello @superm1 ·
          Developer

          If you can please also apply https://gitlab.freedesktop.org/drm/amd/uploads/f5dbaa48d67b6214a723a145d251fff8/s2idle_test.diff and see if it helps. It's not the exact same failure mode, but it could be related.

        • Please register or sign in to reply
      • Marco Laux mentioned in issue #1710 (closed)

        mentioned in issue #1710 (closed)

      • Mario Limonciello added s0ix label

        added s0ix label

      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      Please register or sign in to reply