Skip to content
Snippets Groups Projects

[amdgpu] Steam Deck *ERROR* ring sdma0 timeout

    • Open Issue created by uncle jack @unclejack

    Brief summary of the problem:

    The Steam Deck's gamescope session crashes with a ring sdma0 timeout error. The game was sometimes frozen on the screen for a while. Opening the Steam OS menus and clicking the buttons again sometimes stopped the game. The gamescope session would recover later after some timeout.

    Reproducing the issue was quite difficult. It would occur once every two or three days or after 3-5 hours of use. I was able to find a way to reproduce this after thinking of one particular scenario in which this crash occurred. This involved picking up the Steam Deck from its case, booting it (not waking it from standby) and starting the game. Shutting down the Steam Deck, turning it on and trying to start the game seemed to be a good way to trigger the bug. I was able to trigger it once every 3-10 attempts.

    amdgpu crashed after going through this sequence of steps several times. This seemed to be a good way to trigger this crash.

    I've grabbed Valve's kernel sources to build a kernel for Steam OS with several patches applied to it. The final kernel which still runs into this failure quite often included the following patches:

    The provided sdma0 ring dump was grabbed while running the fully patched kernel. Please let me know if you wish me to send you the kernel package I've built.

    USB devices connected to the Steam Deck didn't make a difference. amdgpu crashed with a keyboard connected, with the Dock and without any USB devices connected to it.

    The crash occurred on battery and while connected to the Valve provided charger. The presence or absence of the charger didn't seem to make a difference.

    Reproducing the bug after a cold boot seems to be easier. amdgpu didn't seem to crash again after the first initial crash after a cold boot. Rebooting may also not help with reproducing the bug. It's much easier to crash amdgpu after a cold boot than after a reboot. Crashing it after running a game for a while may be possible.

    Something else which may be relevant is that this crash occurred with the beta Steam OS 3.5.0 on a different Steam Deck unit. One such crash on that device had as a side effect persistent corruption of the image. The OS still ran properly. The alternating bars pattern went away after a reboot and a power cycle. The reboot on its own didn't seem to help that unit.

    Hardware description:

    • HW: Steam Deck LCD
    • CPU: Steam Deck's APU
    • GPU: Steam Deck's RDNA2 iGPU
    • System Memory: 16 GB
    • Display(s): Steam Deck's integrated LCD display
    • Type of Display Connection: -

    System information:

    • Distro name and Version: SteamOS 3.5.13
    • Kernel version: 6.1.52.valve14
    • Custom kernel: 6.1.52.valve14 (with the mentioned patches)

    How to reproduce the issue:

    1. grab a Steam Deck with Steam OS 3.5.13
    2. set up the password for the deck user & enable ssh
    3. install Elden Ring
    4. configure the game to use Proton Experimental (it's what I was using, probably doesn't matter)
    5. start the game once or twice
    6. make sure it runs
    7. shut down the Steam Deck
    8. turn on the Steam Deck
    9. start Elden Ring from the gamescope session without running any other app/game
    10. if it crashes after loading the game, the screen freezes with an in game image, sometimes crashes to a black screen when gamescope goes down
    11. if it crashes while loading, the screen freezes on a black screen (with a frame counter or complete mangohud overlay if enabled)
    12. connect via SSH to check if amdgpu has crashed
    13. repeat from step 6 if it didn't crash at all

    Attached files:

    amdgpu_ring_sdma0

    Log files (for system lockups / game freezes / crashes)

    Log from the patched kernel (identical to the unpatched one)

    [   20.158985] [drm] Failed to add display topology, DTM TA is not initialized.
[   49.163676] [drm] Failed to add display topology, DTM TA is not initialized.
[   81.648713] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0 timeout, signaled seq=5623, emitted seq=5627
[   81.649066] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process  pid 0 thread  pid 0
[   81.649297] amdgpu 0000:04:00.0: amdgpu: GPU reset begin!
[   81.742639] amdgpu 0000:04:00.0: amdgpu: MODE2 reset
[   81.752816] amdgpu 0000:04:00.0: amdgpu: GPU reset succeeded, trying to resume

    These are the logs generated when recovery was disabled:

    [   42.268799] [drm] Failed to add display topology, DTM TA is not initialized.
[   69.609641] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0 timeout, signaled seq=5518, emitted seq=5522
[   69.610193] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process  pid 0 thread  pid 0
[   69.610702] amdgpu 0000:04:00.0: amdgpu: GPU recovery disabled.

    There are no other relevant log messages, errors related to amdgpu or stacktraces. The messages which are missing are the usual amdgpu initialization messages for the Steam Deck's RDNA2 iGPU.

    Designs ...

    Child items 0

    • No child items are currently assigned. Use child items to break down this issue into smaller parts.

    Activity

    • uncle jack mentioned in issue #2220 (closed)

      mentioned in issue #2220 (closed)

    • Mario Limonciello added Vangogh sdma labels

      added Vangogh sdma labels

    • Mario Limonciello added hang/freeze label

      added hang/freeze label

    • Mario Limonciello
      Mario Limonciello @superm1 ·
      Developer

      CC @pyuan

      • Marco Rodolfi
        Marco Rodolfi @RodoMa92 ·

        Hit this again today while browsing the web and using PDFs software, log below (sadly still completely useless)

        gen 22 13:11:18 DeckDiMarco kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0 timeout, signaled seq=139119, emitted seq=139121
gen 22 13:11:18 DeckDiMarco kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process  pid 0 thread  pid 0
gen 22 13:11:18 DeckDiMarco kernel: amdgpu 0000:04:00.0: amdgpu: GPU reset begin!
gen 22 13:11:18 DeckDiMarco kernel: amdgpu 0000:04:00.0: amdgpu: MODE2 reset
gen 22 13:11:18 DeckDiMarco kernel: amdgpu 0000:04:00.0: amdgpu: GPU reset succeeded, trying to resume
gen 22 13:11:18 DeckDiMarco kernel: [drm] PCIE GART of 1024M enabled (table at 0x000000F43FC00000).
gen 22 13:11:18 DeckDiMarco kernel: [drm] PSP is resuming...
gen 22 13:11:18 DeckDiMarco kernel: [drm] reserve 0xa00000 from 0xf43e000000 for PSP TMR
gen 22 13:11:19 DeckDiMarco kernel: amdgpu 0000:04:00.0: amdgpu: SMU is resuming...
gen 22 13:11:19 DeckDiMarco kernel: amdgpu 0000:04:00.0: amdgpu: SMU is resumed successfully!
gen 22 13:11:19 DeckDiMarco kernel: [drm] DMUB hardware initialized: version=0x0300000A
gen 22 13:11:19 DeckDiMarco kernel: [drm] Failed to add display topology, DTM TA is not initialized.
gen 22 13:11:19 DeckDiMarco kernel: [drm] kiq ring mec 2 pipe 1 q 0
gen 22 13:11:19 DeckDiMarco kernel: [drm] VCN decode and encode initialized successfully(under DPG Mode).
gen 22 13:11:19 DeckDiMarco kernel: [drm] JPEG decode initialized successfully.
gen 22 13:11:19 DeckDiMarco kernel: amdgpu 0000:04:00.0: amdgpu: ring gfx_0.0.0 uses VM inv eng 0 on hub 0
gen 22 13:11:19 DeckDiMarco kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 1 on hub 0
gen 22 13:11:19 DeckDiMarco kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 4 on hub 0
gen 22 13:11:19 DeckDiMarco kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 5 on hub 0
gen 22 13:11:19 DeckDiMarco kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 6 on hub 0
gen 22 13:11:19 DeckDiMarco kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 7 on hub 0
gen 22 13:11:19 DeckDiMarco kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 8 on hub 0
gen 22 13:11:19 DeckDiMarco kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 9 on hub 0
gen 22 13:11:19 DeckDiMarco kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 10 on hub 0
gen 22 13:11:19 DeckDiMarco kernel: amdgpu 0000:04:00.0: amdgpu: ring kiq_0.2.1.0 uses VM inv eng 11 on hub 0
gen 22 13:11:19 DeckDiMarco kernel: amdgpu 0000:04:00.0: amdgpu: ring sdma0 uses VM inv eng 12 on hub 0
gen 22 13:11:19 DeckDiMarco kernel: amdgpu 0000:04:00.0: amdgpu: ring vcn_dec_0 uses VM inv eng 0 on hub 8
gen 22 13:11:19 DeckDiMarco kernel: amdgpu 0000:04:00.0: amdgpu: ring vcn_enc_0.0 uses VM inv eng 1 on hub 8
gen 22 13:11:19 DeckDiMarco kernel: amdgpu 0000:04:00.0: amdgpu: ring vcn_enc_0.1 uses VM inv eng 4 on hub 8
gen 22 13:11:19 DeckDiMarco kernel: amdgpu 0000:04:00.0: amdgpu: ring jpeg_dec uses VM inv eng 5 on hub 8
gen 22 13:11:19 DeckDiMarco kernel: amdgpu 0000:04:00.0: amdgpu: recover vram bo from shadow start
gen 22 13:11:19 DeckDiMarco kernel: amdgpu 0000:04:00.0: amdgpu: recover vram bo from shadow done
gen 22 13:11:19 DeckDiMarco kernel: amdgpu 0000:04:00.0: amdgpu: GPU reset(1) succeeded!
gen 22 13:11:19 DeckDiMarco kernel: [drm] Skip scheduling IBs!

        Regardless of the successful state reported from the driver, the experience differs (whole desktop frozen after the usual flash, only full reboot made the panel come back to life).

        6.6.11-201.fsync.fc39.x86_64

        Edited by Marco Rodolfi
      • Marco Rodolfi
        Marco Rodolfi @RodoMa92 ·

        At this point I'm sure that #2816 (closed) is the same issue as reported here. In my case I can only reproduce on light load only, not sure if @unclejack has the same behaviour.

      • Marco Rodolfi
        Marco Rodolfi @RodoMa92 ·

        And since the issue can be reproduced on both 6.1 LTS and 6.6 current to me is quite evident that the issues lies in firmware and not on the open source driver part.

      • Mario Limonciello
        Mario Limonciello @superm1 ·
        Developer

        quite evident that the issues lies in firmware and not on the open source driver part.

        This might be true; but it could also be a hardware issue as well. In either case it's plausible to workaround in the driver code once it's understood and root caused. That's what we did in the Rembrandt SDMA workaround for example. It's too bad they're not the same root cause :cry:.

      • uncle jack
        uncle jack @unclejack ·
        Author

        @RodoMa92: The driver crashed for me while playing the game or when starting the same game. That was my somewhat reliable way to reproduce the crash. It appears to be somehow related to the state of the hardware. It gave me the impression that it's easier to trigger after a cold boot.

        I don't use the Steam Deck for other activities. This is probably why it hasn't crashed in other scenarios so far.

        This bug has taken quite a bit of my time with testing various things, trying to isolate it and writing it all down. AMD and Valve pay people to do this work. I can't continue to do these tests forever.

        My main focus now is to wrap up the testing for the other ticket related to the RX 7800 XT bugs and to figure out if that GPU board is just RMA material.

      • Please register or sign in to reply
    • Marco Rodolfi
      Marco Rodolfi @RodoMa92 ·

      One final observation is that voltage doesn't seems to be the cause (while undervolting the GPU), since I could reproduce it both with it and without it in my experience as of now.

      • Marco Rodolfi
        Marco Rodolfi @RodoMa92 ·

        Disclaimer: I'm not responsible for blowing up hardware if you want to follow the steps below: playing with voltages is risky if you do not know what you are doing

        I might have got to the bottom of at least MY issue with this: it seems to be related to CPU undervolting (I completely forgot about it since I've set a year and a half ago, never crashed before). I have reduced the amount applied using ryzenadj (literally one unit) and for now I haven't been able to reproduce it again (and I had this issue like 4 times these two days). A little bit too early to be sure, tho.

        @unclejack Do you have any CPU undervolting applied that you also forgot to have? If yes, try to push it up a couple of notches and see if this happens again. If not, AMD might need to tweak the base voltage to the CPU itself. You could try to use ryzenadj and try to apply a positive voltage curve (like 1 or 2 increments that should translate to a gain of 3x(n) to 5x(n) mV to the CPU) to see if this goes away. Do this only if you feel confident enough (you CAN potentially blow up hardware if you make a mistake and the SMU doesn't have failsafes in, so you are responsible for eventual issues arising from this change).

        Might be worth to AMD test for undervolting it until a crash, then step back a couple of notches until it look stable and see if the issue above is reproduced. This would confirm that at least for this ASIC undervolting the CPU makes the sdma0 ring mad.

        Edited by Marco Rodolfi
      • uncle jack
        uncle jack @unclejack ·
        Author

        I've run into these crashes with two Steam Deck units so far. The first one was sent off to RMA after displaying alternating bars on the screen until it was power cycled and rebooted several times. The second one is the current one which ran into these crashes over the last two months.

        The firmware/BIOS settings are all at 0 for all available voltage offsets. This Steam Deck hasn't been opened (the previous one either). All firmware settings are as they were configured by Valve or the company to which they outsource the development of this firmware. This means that IOMMU is off, fTPM is enabled and USB is set to XHCI, not dual role. This was actually the first time I had to enter the firmware settings on this unit.

        The Steam OS running on the Steam Deck is also used without any third party tools which would control power usage or adjust voltages. The only modification made to the OS is the installation of the kernel package. This was the custom kernel package before and now the binary one from Valve.

        I wasn't able to reproduce the failures so far with the latest kernel from Valve. It's likely that it works around the hardware bug to avoid the crash. It seems to do so at least in a very particular scenario.

        My suggestion would be to try the new kernel from Valve or grab those patches to apply them on top of a different kernel which is used by your distro.

      • Marco Rodolfi
        Marco Rodolfi @RodoMa92 ·

        So it might be just sheer luck that I haven't hit them again on my end. Good to know. I'll try to forward them to my distro mantainer for inclusion, assuming they aren't yet merged in mainline.

      • uncle jack
        uncle jack @unclejack ·
        Author

        Those commits should be used to build test kernel packages. The packages could be used in a test build of the distro. It doesn't appear to be something everyone should use right now. It's probably a good idea to let those who encounter the amdgpu crashes test the special build with these patches. That's probably the best way to do this.

      • Marco Rodolfi
        Marco Rodolfi @RodoMa92 ·

        Yeah, I asked them to include in the testing branch for now. Let me know if you can reproduce the crashes again.

      • Please register or sign in to reply
    • Marco Rodolfi mentioned in issue #2816 (closed)

      mentioned in issue #2816 (closed)

    • Marco Rodolfi
      Marco Rodolfi @RodoMa92 ·

      Crashed again with the same log and the patches mentioned above, so it's just extremely random. I'll fully disable any undervolting to be sure, tho. it happened while playing a youtube video using freetube, but this time the driver properly recovered for once.

      Any additional steps I can try, AMD?

      • Vitor Machado
        Vitor Machado @vittau ·

        Just had a similar crash on my Steam Deck, while on Desktop Mode. I was using battery, which had a ~94% charge. The Deck is on the Stable branch.

        I had just booted up the Deck from a cold start, I was using it for 10 minutes or so. Then the Deck froze for a couple seconds, then a black screen, back for a couple seconds and froze to black screen again, back for a couple seconds one more time, freeze to black. And finally this time it rebooted into the "Verifying installation" screen.

        The logs start with the sdma0 timeout, and then it did the gfx_0.0.0 timeout error as well.

        I had similar issues twice before, but the other two times I was on Game Mode and I only got the gfx_0.0.0 timeout error. This is the first time I get the sdma0 timeout one.

        Here is my post from the previous crash: #2477 (comment 2246734)

        Since it only happened 3 times I don't have much statistical significance, but in all 3 cases I was on battery, doing very lightweight activities. Just yesterday I played Persona 3 Reload for over 5h while docked, without any issues. Also, these problems started not long after the SteamOS 3.5 update, which may or may not be a coincidence.

        I attach below the most relevant part of journalctl of this crash:

        Feb 04 13:22:52 steamdeck kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0 timeout, signaled seq=3015, emitted seq=3017
Feb 04 13:22:52 steamdeck kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process  pid 0 thread  pid 0
Feb 04 13:22:52 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: GPU reset begin!
Feb 04 13:22:52 steamdeck fancontrol.py[577]: Traceback (most recent call last):
Feb 04 13:22:52 steamdeck fancontrol.py[577]:   File "/usr/share/jupiter-fan-control/fancontrol.py", line 542, in <module>
Feb 04 13:22:52 steamdeck fancontrol.py[577]:     controller.loop_control()
Feb 04 13:22:52 steamdeck fancontrol.py[577]:   File "/usr/share/jupiter-fan-control/fancontrol.py", line 486, in loop_control
Feb 04 13:22:52 steamdeck fancontrol.py[577]:     self.loop_read_sensors()
Feb 04 13:22:52 steamdeck fancontrol.py[577]:   File "/usr/share/jupiter-fan-control/fancontrol.py", line 452, in loop_read_sensors
Feb 04 13:22:52 steamdeck fancontrol.py[577]:     self.power_sensor.get_avg_value()
Feb 04 13:22:52 steamdeck fancontrol.py[577]:   File "/usr/share/jupiter-fan-control/fancontrol.py", line 356, in get_avg_value
Feb 04 13:22:52 steamdeck fancontrol.py[577]:     self.values.append(self.get_value())
Feb 04 13:22:52 steamdeck fancontrol.py[577]:                        ^^^^^^^^^^^^^^^^
Feb 04 13:22:52 steamdeck fancontrol.py[577]:   File "/usr/share/jupiter-fan-control/fancontrol.py", line 351, in get_value
Feb 04 13:22:52 steamdeck fancontrol.py[577]:     self.value = int(f.read().strip()) / 1000000
Feb 04 13:22:52 steamdeck fancontrol.py[577]:                      ^^^^^^^^
Feb 04 13:22:52 steamdeck fancontrol.py[577]: PermissionError: [Errno 1] Operation not permitted
Feb 04 13:22:52 steamdeck systemd[1]: jupiter-fan-control.service: Main process exited, code=exited, status=1/FAILURE
Feb 04 13:22:52 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: MODE2 reset
Feb 04 13:22:52 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: GPU reset succeeded, trying to resume
Feb 04 13:22:52 steamdeck kernel: [drm] PCIE GART of 1024M enabled (table at 0x000000F4FFC00000).
Feb 04 13:22:52 steamdeck kernel: [drm] PSP is resuming...
Feb 04 13:22:52 steamdeck (udev-worker)[3965]: devcd1: Process 'cat /sys/devices/virtual/devcoredump/devcd1/data > /var/lib/steamos-log-submitter/pending/devcoredump/4482' failed with exit code 1.
Feb 04 13:22:52 steamdeck kernel: [drm] reserve 0xa00000 from 0xf4fe000000 for PSP TMR
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: SMU is resuming...
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: SMU is resumed successfully!
Feb 04 13:22:53 steamdeck kernel: [drm] DMUB hardware initialized: version=0x0300000A
Feb 04 13:22:53 steamdeck kernel: [drm] Failed to add display topology, DTM TA is not initialized.
Feb 04 13:22:53 steamdeck kernel: [drm] kiq ring mec 2 pipe 1 q 0
Feb 04 13:22:53 steamdeck kernel: [drm] VCN decode and encode initialized successfully(under DPG Mode).
Feb 04 13:22:53 steamdeck kernel: [drm] JPEG decode initialized successfully.
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring gfx_0.0.0 uses VM inv eng 0 on hub 0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 1 on hub 0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 4 on hub 0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 5 on hub 0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 6 on hub 0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 7 on hub 0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 8 on hub 0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 9 on hub 0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 10 on hub 0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring kiq_0.2.1.0 uses VM inv eng 11 on hub 0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring sdma0 uses VM inv eng 12 on hub 0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring vcn_dec_0 uses VM inv eng 0 on hub 8
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring vcn_enc_0.0 uses VM inv eng 1 on hub 8
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring vcn_enc_0.1 uses VM inv eng 4 on hub 8
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring jpeg_dec uses VM inv eng 5 on hub 8
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: recover vram bo from shadow start
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: recover vram bo from shadow done
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: GPU reset(1) succeeded!
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:5 pasid:32782, for process steamwebhelper pid 2925 thread steamwebhe:cs0 pid 2927)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:   in page starting at address 0x0000800106806000 from client 0x1b (UTCL2)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00501031
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          Faulty UTCL2 client ID: TCP (0x8)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MORE_FAULTS: 0x1
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          WALKER_ERROR: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          PERMISSION_FAULTS: 0x3
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MAPPING_ERROR: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          RW: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:5 pasid:32782, for process steamwebhelper pid 2925 thread steamwebhe:cs0 pid 2927)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:   in page starting at address 0x0000800106804000 from client 0x1b (UTCL2)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00000000
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          Faulty UTCL2 client ID: CB/DB (0x0)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MORE_FAULTS: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          WALKER_ERROR: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          PERMISSION_FAULTS: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MAPPING_ERROR: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          RW: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:5 pasid:32782, for process steamwebhelper pid 2925 thread steamwebhe:cs0 pid 2927)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:   in page starting at address 0x0000800106800000 from client 0x1b (UTCL2)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00000000
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          Faulty UTCL2 client ID: CB/DB (0x0)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MORE_FAULTS: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          WALKER_ERROR: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          PERMISSION_FAULTS: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MAPPING_ERROR: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          RW: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:5 pasid:32782, for process steamwebhelper pid 2925 thread steamwebhe:cs0 pid 2927)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:   in page starting at address 0x0000800106807000 from client 0x1b (UTCL2)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00000000
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          Faulty UTCL2 client ID: CB/DB (0x0)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MORE_FAULTS: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          WALKER_ERROR: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          PERMISSION_FAULTS: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MAPPING_ERROR: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          RW: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:5 pasid:32782, for process steamwebhelper pid 2925 thread steamwebhe:cs0 pid 2927)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:   in page starting at address 0x0000800106802000 from client 0x1b (UTCL2)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00000000
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          Faulty UTCL2 client ID: CB/DB (0x0)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MORE_FAULTS: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          WALKER_ERROR: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          PERMISSION_FAULTS: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MAPPING_ERROR: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          RW: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:5 pasid:32782, for process steamwebhelper pid 2925 thread steamwebhe:cs0 pid 2927)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:   in page starting at address 0x0000800106805000 from client 0x1b (UTCL2)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00000000
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          Faulty UTCL2 client ID: CB/DB (0x0)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MORE_FAULTS: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          WALKER_ERROR: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          PERMISSION_FAULTS: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MAPPING_ERROR: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          RW: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:5 pasid:32782, for process steamwebhelper pid 2925 thread steamwebhe:cs0 pid 2927)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:   in page starting at address 0x0000800106803000 from client 0x1b (UTCL2)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00000000
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          Faulty UTCL2 client ID: CB/DB (0x0)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MORE_FAULTS: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          WALKER_ERROR: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          PERMISSION_FAULTS: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MAPPING_ERROR: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          RW: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:5 pasid:32782, for process steamwebhelper pid 2925 thread steamwebhe:cs0 pid 2927)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:   in page starting at address 0x0000800106800000 from client 0x1b (UTCL2)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00000000
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          Faulty UTCL2 client ID: CB/DB (0x0)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MORE_FAULTS: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          WALKER_ERROR: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          PERMISSION_FAULTS: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MAPPING_ERROR: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          RW: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:5 pasid:32782, for process steamwebhelper pid 2925 thread steamwebhe:cs0 pid 2927)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:   in page starting at address 0x0000800106801000 from client 0x1b (UTCL2)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00000000
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          Faulty UTCL2 client ID: CB/DB (0x0)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MORE_FAULTS: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          WALKER_ERROR: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          PERMISSION_FAULTS: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MAPPING_ERROR: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          RW: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:5 pasid:32782, for process steamwebhelper pid 2925 thread steamwebhe:cs0 pid 2927)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:   in page starting at address 0x0000800106801000 from client 0x1b (UTCL2)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00000000
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          Faulty UTCL2 client ID: CB/DB (0x0)
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MORE_FAULTS: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          WALKER_ERROR: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          PERMISSION_FAULTS: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MAPPING_ERROR: 0x0
Feb 04 13:22:53 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          RW: 0x0
Feb 04 13:23:02 steamdeck systemd[1]: jupiter-fan-control.service: State 'stop-post' timed out. Terminating.
Feb 04 13:23:02 steamdeck systemd[1]: jupiter-fan-control.service: Control process exited, code=killed, status=15/TERM
Feb 04 13:23:02 steamdeck systemd[1]: jupiter-fan-control.service: Failed with result 'exit-code'.
Feb 04 13:23:03 steamdeck kernel: gmc_v10_0_process_interrupt: 92 callbacks suppressed
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:5 pasid:32782, for process steamwebhelper pid 2925 thread steamwebhe:cs0 pid 2927)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:   in page starting at address 0x0000800106804000 from client 0x1b (UTCL2)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00501031
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          Faulty UTCL2 client ID: TCP (0x8)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MORE_FAULTS: 0x1
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          WALKER_ERROR: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          PERMISSION_FAULTS: 0x3
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MAPPING_ERROR: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          RW: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:5 pasid:32782, for process steamwebhelper pid 2925 thread steamwebhe:cs0 pid 2927)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:   in page starting at address 0x0000800106801000 from client 0x1b (UTCL2)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00501031
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          Faulty UTCL2 client ID: TCP (0x8)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MORE_FAULTS: 0x1
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          WALKER_ERROR: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          PERMISSION_FAULTS: 0x3
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MAPPING_ERROR: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          RW: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:5 pasid:32782, for process steamwebhelper pid 2925 thread steamwebhe:cs0 pid 2927)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:   in page starting at address 0x000080010681c000 from client 0x1b (UTCL2)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00501031
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          Faulty UTCL2 client ID: TCP (0x8)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MORE_FAULTS: 0x1
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          WALKER_ERROR: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          PERMISSION_FAULTS: 0x3
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MAPPING_ERROR: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          RW: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:5 pasid:32782, for process steamwebhelper pid 2925 thread steamwebhe:cs0 pid 2927)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:   in page starting at address 0x0000800106805000 from client 0x1b (UTCL2)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00501031
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          Faulty UTCL2 client ID: TCP (0x8)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MORE_FAULTS: 0x1
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          WALKER_ERROR: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          PERMISSION_FAULTS: 0x3
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MAPPING_ERROR: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          RW: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:5 pasid:32782, for process steamwebhelper pid 2925 thread steamwebhe:cs0 pid 2927)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:   in page starting at address 0x000080010681d000 from client 0x1b (UTCL2)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00501031
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          Faulty UTCL2 client ID: TCP (0x8)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MORE_FAULTS: 0x1
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          WALKER_ERROR: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          PERMISSION_FAULTS: 0x3
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MAPPING_ERROR: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          RW: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:5 pasid:32782, for process steamwebhelper pid 2925 thread steamwebhe:cs0 pid 2927)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:   in page starting at address 0x0000800106806000 from client 0x1b (UTCL2)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00501031
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          Faulty UTCL2 client ID: TCP (0x8)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MORE_FAULTS: 0x1
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          WALKER_ERROR: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          PERMISSION_FAULTS: 0x3
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MAPPING_ERROR: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          RW: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:5 pasid:32782, for process steamwebhelper pid 2925 thread steamwebhe:cs0 pid 2927)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:   in page starting at address 0x0000800106802000 from client 0x1b (UTCL2)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00501031
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          Faulty UTCL2 client ID: TCP (0x8)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MORE_FAULTS: 0x1
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          WALKER_ERROR: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          PERMISSION_FAULTS: 0x3
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MAPPING_ERROR: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          RW: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:5 pasid:32782, for process steamwebhelper pid 2925 thread steamwebhe:cs0 pid 2927)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:   in page starting at address 0x0000800106807000 from client 0x1b (UTCL2)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00501031
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          Faulty UTCL2 client ID: TCP (0x8)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MORE_FAULTS: 0x1
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          WALKER_ERROR: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          PERMISSION_FAULTS: 0x3
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MAPPING_ERROR: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          RW: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:5 pasid:32782, for process steamwebhelper pid 2925 thread steamwebhe:cs0 pid 2927)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:   in page starting at address 0x0000800106803000 from client 0x1b (UTCL2)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00501031
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          Faulty UTCL2 client ID: TCP (0x8)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MORE_FAULTS: 0x1
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          WALKER_ERROR: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          PERMISSION_FAULTS: 0x3
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MAPPING_ERROR: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          RW: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:5 pasid:32782, for process steamwebhelper pid 2925 thread steamwebhe:cs0 pid 2927)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:   in page starting at address 0x0000800106804000 from client 0x1b (UTCL2)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00501031
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          Faulty UTCL2 client ID: TCP (0x8)
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MORE_FAULTS: 0x1
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          WALKER_ERROR: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          PERMISSION_FAULTS: 0x3
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          MAPPING_ERROR: 0x0
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu:          RW: 0x0
Feb 04 13:23:03 steamdeck kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx_0.0.0 timeout, signaled seq=44038, emitted seq=44040
Feb 04 13:23:03 steamdeck kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process steamwebhelper pid 2925 thread steamwebhe:cs0 pid 2927
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: GPU reset begin!
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: MODE2 reset
Feb 04 13:23:03 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: GPU reset succeeded, trying to resume
Feb 04 13:23:03 steamdeck kernel: [drm] PCIE GART of 1024M enabled (table at 0x000000F4FFC00000).
Feb 04 13:23:03 steamdeck kernel: [drm] PSP is resuming...
Feb 04 13:23:03 steamdeck kernel: [drm] reserve 0xa00000 from 0xf4fe000000 for PSP TMR
Feb 04 13:23:03 steamdeck systemd[1]: jupiter-fan-control.service: Scheduled restart job, restart counter is at 1.
Feb 04 13:23:03 steamdeck systemd[1]: Stopped Jupiter fan control.
Feb 04 13:23:03 steamdeck systemd[1]: Started Jupiter fan control.
Feb 04 13:23:03 steamdeck steam[2846]: src/clientdll/steamengine.cpp (2647) : Assertion Failed: CSteamEngine::BMainLoop appears to have stalled > 15 seconds without event signalled
Feb 04 13:23:03 steamdeck steam[2846]: src/clientdll/steamengine.cpp (2647) : Assertion Failed: CSteamEngine::BMainLoop appears to have stalled > 15 seconds without event signalled
Feb 04 13:23:04 steamdeck steam[3987]: assert_20240204132303_39.dmp[3987]: Uploading dump (out-of-process)
Feb 04 13:23:04 steamdeck steam[3987]: /tmp/dumps/assert_20240204132303_39.dmp
Feb 04 13:23:04 steamdeck assert_20240204132303_39.dmp[3987]: Uploading dump (out-of-process)
                                                              /tmp/dumps/assert_20240204132303_39.dmp
Feb 04 13:23:04 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: SMU is resuming...
Feb 04 13:23:04 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: SMU is resumed successfully!
Feb 04 13:23:04 steamdeck kernel: [drm] DMUB hardware initialized: version=0x0300000A
Feb 04 13:23:04 steamdeck kernel: [drm] Failed to add display topology, DTM TA is not initialized.
Feb 04 13:23:04 steamdeck kernel: [drm] kiq ring mec 2 pipe 1 q 0
Feb 04 13:23:04 steamdeck kernel: [drm] VCN decode and encode initialized successfully(under DPG Mode).
Feb 04 13:23:04 steamdeck kernel: [drm] JPEG decode initialized successfully.
Feb 04 13:23:04 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring gfx_0.0.0 uses VM inv eng 0 on hub 0
Feb 04 13:23:04 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 1 on hub 0
Feb 04 13:23:04 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 4 on hub 0
Feb 04 13:23:04 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 5 on hub 0
Feb 04 13:23:04 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 6 on hub 0
Feb 04 13:23:04 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 7 on hub 0
Feb 04 13:23:04 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 8 on hub 0
Feb 04 13:23:04 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 9 on hub 0
Feb 04 13:23:04 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 10 on hub 0
Feb 04 13:23:04 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring kiq_0.2.1.0 uses VM inv eng 11 on hub 0
Feb 04 13:23:04 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring sdma0 uses VM inv eng 12 on hub 0
Feb 04 13:23:04 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring vcn_dec_0 uses VM inv eng 0 on hub 8
Feb 04 13:23:04 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring vcn_enc_0.0 uses VM inv eng 1 on hub 8
Feb 04 13:23:04 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring vcn_enc_0.1 uses VM inv eng 4 on hub 8
Feb 04 13:23:04 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: ring jpeg_dec uses VM inv eng 5 on hub 8
Feb 04 13:23:04 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: recover vram bo from shadow start
Feb 04 13:23:04 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: recover vram bo from shadow done
Feb 04 13:23:04 steamdeck kernel: [drm] Skip scheduling IBs!
Feb 04 13:23:04 steamdeck kernel: [drm] Skip scheduling IBs!
Feb 04 13:23:04 steamdeck kernel: amdgpu 0000:04:00.0: amdgpu: GPU reset(3) succeeded!
Feb 04 13:23:05 steamdeck fancontrol.py[3978]: loaded critical temp from SSD hwmon: 87.85
Feb 04 13:23:05 steamdeck fancontrol.py[3978]: jupiter-fan-control started successfully.
Feb 04 13:23:07 steamdeck steam[3987]: assert_20240204132303_39.dmp[3987]: Finished uploading minidump (out-of-process): success = yes
Feb 04 13:23:07 steamdeck steam[3987]: assert_20240204132303_39.dmp[3987]: response: CrashID=bp-c5c9104d-74d6-4523-b71d-445962240204
Feb 04 13:23:07 steamdeck steam[3987]: assert_20240204132303_39.dmp[3987]: file ''/tmp/dumps/assert_20240204132303_39.dmp'', upload yes: ''CrashID=bp-c5c9104d-74d6-4523-b71d-445962240204''
Feb 04 13:23:07 steamdeck assert_20240204132303_39.dmp[3987]: Finished uploading minidump (out-of-process): success = yes
Feb 04 13:23:07 steamdeck assert_20240204132303_39.dmp[3987]: response: CrashID=bp-c5c9104d-74d6-4523-b71d-445962240204
Feb 04 13:23:07 steamdeck assert_20240204132303_39.dmp[3987]: file ''/tmp/dumps/assert_20240204132303_39.dmp'', upload yes: ''CrashID=bp-c5c9104d-74d6-4523-b71d-445962240204''
Feb 04 13:23:14 steamdeck kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx_0.0.0 timeout, but soft recovered
Feb 04 13:23:14 steamdeck ark[3898]: The X11 connection broke (error 1). Did the X11 server die?
Feb 04 13:23:14 steamdeck dolphin[3864]: The X11 connection broke (error 1). Did the X11 server die?
Feb 04 13:23:14 steamdeck kded5[2358]: X connection to :0 broken (explicit kill or server shutdown).
Feb 04 13:23:14 steamdeck plasmashell[2384]: X connection to :0 broken (explicit kill or server shutdown).
Feb 04 13:23:14 steamdeck kimpanel-ibus-panel[2613]: The X11 connection broke (error 1). Did the X11 server die?
Feb 04 13:23:14 steamdeck kglobalaccel5[2386]: The X11 connection broke (error 1). Did the X11 server die?
Feb 04 13:23:14 steamdeck plasmashell[2384]: The X11 connection broke: I/O error (code 1)
Feb 04 13:23:14 steamdeck kaccess[2581]: X connection to :0 broken (explicit kill or server shutdown).
Feb 04 13:23:14 steamdeck kded5[2358]: The X11 connection broke: I/O error (code 1)
Feb 04 13:23:14 steamdeck xdg-desktop-portal-kde[2436]: The X11 connection broke (error 1). Did the X11 server die?
Feb 04 13:23:14 steamdeck kdeconnectd[2572]: The X11 connection broke (error 1). Did the X11 server die?
Feb 04 13:23:14 steamdeck kaccess[2581]: The X11 connection broke: I/O error (code 1)
Feb 04 13:23:14 steamdeck polkit-kde-authentication-agent-1[2434]: The X11 connection broke (error 1). Did the X11 server die?
Feb 04 13:23:14 steamdeck kglobalaccel5[2386]: The X11 connection broke: I/O error (code 1)
Feb 04 13:23:14 steamdeck kglobalaccel5[2386]: XIO:  fatal IO error 25 (Inappropriate ioctl for device) on X server ":0"
Feb 04 13:23:14 steamdeck kglobalaccel5[2386]:       after 8 requests (8 known processed) with 0 events remaining.
Feb 04 13:23:14 steamdeck org_kde_powerdevil[2435]: The X11 connection broke (error 1). Did the X11 server die?
Feb 04 13:23:14 steamdeck at-spi-bus-launcher[2704]: X connection to :0 broken (explicit kill or server shutdown).
Feb 04 13:23:14 steamdeck kwin_x11[2359]: kwin_scene_opengl: A graphics reset not attributable to the current GL context occurred.
Feb 04 13:23:14 steamdeck DiscoverNotifier[2583]: The X11 connection broke (error 1). Did the X11 server die?
Feb 04 13:23:14 steamdeck ksmserver[2356]: The X11 connection broke (error 1). Did the X11 server die?
Feb 04 13:23:14 steamdeck kscreen_backend_launcher[2539]: The X11 connection broke (error 1). Did the X11 server die?
Feb 04 13:23:14 steamdeck systemd[887]: plasma-kglobalaccel.service: Main process exited, code=exited, status=1/FAILURE
Feb 04 13:23:14 steamdeck systemd[887]: plasma-kglobalaccel.service: Failed with result 'exit-code'.
Feb 04 13:23:14 steamdeck kactivitymanagerd[2430]: The X11 connection broke (error 1). Did the X11 server die?
Feb 04 13:23:14 steamdeck plasmashell[2384]: QtDBus: cannot relay signals from parent QObject(0x557e257df890 "") unless they are emitted in the object's thread QThread(0x557e246d3bb0 ""). Current thread is QSGRenderThread(0x557e254567a0 "").
Feb 04 13:23:14 steamdeck plasma-discover[3496]: The X11 connection broke (error 1). Did the X11 server die?
Feb 04 13:23:14 steamdeck plasmashell[2384]: QSocketNotifier: Socket notifiers cannot be enabled or disabled from another thread
Feb 04 13:23:14 steamdeck plasmashell[2384]: QBasicTimer::stop: Failed. Possibly trying to stop from a different thread
Feb 04 13:23:14 steamdeck plasmashell[2384]: QBasicTimer::stop: Failed. Possibly trying to stop from a different thread
Feb 04 13:23:14 steamdeck plasmashell[2384]: QBasicTimer::stop: Failed. Possibly trying to stop from a different thread
Feb 04 13:23:14 steamdeck plasmashell[2384]: QBasicTimer::stop: Failed. Possibly trying to stop from a different thread
Feb 04 13:23:14 steamdeck plasmashell[2384]: QBasicTimer::stop: Failed. Possibly trying to stop from a different thread
Feb 04 13:23:14 steamdeck xembedsniproxy[2437]: The X11 connection broke (error 1). Did the X11 server die?
Feb 04 13:23:14 steamdeck gmenudbusmenuproxy[2433]: The X11 connection broke (error 1). Did the X11 server die?
Feb 04 13:23:14 steamdeck systemd[887]: Stopped target plasma-workspace-x11.target.
Feb 04 13:23:14 steamdeck systemd[887]: Stopped target KDE Plasma Workspace.
Feb 04 13:23:14 steamdeck systemd[887]: Stopped target Startup of XDG autostart applications.
Feb 04 13:23:14 steamdeck systemd[887]: Stopping Geoclue Demo agent...
Feb 04 13:23:14 steamdeck systemd[887]: Stopping IBus...
Feb 04 13:23:14 steamdeck systemd[887]: Stopping Accessibility...
Feb 04 13:23:14 steamdeck systemd[887]: Stopping Discover...
Feb 04 13:23:14 steamdeck systemd[887]: Stopping KDE Connect...
Feb 04 13:23:14 steamdeck systemd[887]: Stopping Steam...
Feb 04 13:23:14 steamdeck systemd[887]: Stopping Accessibility services bus...
Feb 04 13:23:14 steamdeck systemd[887]: Stopping flatpak session helper...
Feb 04 13:23:14 steamdeck systemd[887]: Stopping Proxies GTK DBus menus to a Plasma readable format...
Feb 04 13:23:14 steamdeck systemd[887]: Stopping KScreen...
Feb 04 13:23:14 steamdeck systemd[887]: Stopping KDE Window Manager...
Feb 04 13:23:14 steamdeck systemd[887]: Stopping KDE PolicyKit Authentication Agent...
Feb 04 13:23:14 steamdeck systemd[887]: Stopping Xdg Desktop Portal For KDE...
Feb 04 13:23:14 steamdeck systemd[887]: Stopping Handle legacy xembed system tray icons...
Feb 04 13:23:14 steamdeck systemd[887]: Stopping Portal service...
Feb 04 13:23:14 steamdeck systemd[887]: Stopping flatpak document portal service...
Feb 04 13:23:14 steamdeck systemd[887]: Stopping sandboxed app permission store...
Feb 04 13:23:14 steamdeck systemd[887]: Stopping Baloo File Indexer Daemon...
        Edited by Vitor Machado
      • Marco Rodolfi
        Marco Rodolfi @RodoMa92 ·

        Yeah, the trigger (low loads) and the log seems quite close to my experience, so this might be the exact same bug.

      • Vitor Machado
        Vitor Machado @vittau ·

        I really hope this is not a generalized RDNA2 production issue, because I live in Brazil and the Deck is not officially supported here...

      • uncle jack
        uncle jack @unclejack ·
        Author

        @vittau: It's not very likely that your Steam Deck unit has a problem. This appears to be a mix of hardware and software issues which affect many units. My previous Steam Deck had issues with the 3.5 beta/preview. The new unit I'm using also ran into the issue.

        I haven't been able to reproduce this issue for several weeks now with the kernel mentioned above. SteamOS 3.5.14 reverted the kernel to a version without the patches. It still didn't crash. It's likely to happen again at some point.

      • Vitor Machado
        Vitor Machado @vittau ·

        Well, had a crash today with the Deck plugged in to my dock, so there goes my theory of this only happening while using the battery on my unit. However, the circumstances were similar as before: I had just turned on the device a couple minutes before, and it crashed while browsing the Steam interface. This time it blinked a couple times but did not reboot by itself, I had to force shutdown with the power button. I'm still on the Stable branch.

        After that, I proceeded to play Persona 3 Reload for many hours without any issues.

      • Vitor Machado
        Vitor Machado @vittau ·

        After all these months, the crash just happened again on my LCD Deck. I thought this was completely fixed, but apparently not. I'm on the Stable branch, kernel is 6.1.52-valve16-1-neptune-61.

      • Natalie Vock
        Natalie Vock @pixelcluster ·

        The workaround hasn't landed in Stable yet. I think 3.6 is already in Preview and should contain a 6.5.x kernel which also includes the workaround.

      • Vitor Machado
        Vitor Machado @vittau ·

        So I was lucky all this time? Interesting, it's been so long that I thought some workaround had landed. Was there some previous simpler workaround that reduced the incidence rate?

      • Natalie Vock
        Natalie Vock @pixelcluster ·

        This issue is pretty timing-sensitive, so even unrelated changes might cause some random thing to take longer/shorter, which would affect how often you see this. Perhaps this is why you didn't see the issue for a long time - the workaround in Preview should fix it for good.

      • Please register or sign in to reply
    • Marco Rodolfi mentioned in issue #3256

      mentioned in issue #3256

    • Marco Rodolfi
      Marco Rodolfi @RodoMa92 ·

      Still randomly happening on 6.7.9-204.fsync.fc39.x86_64 while idle.

      • theofficialgman
        theofficialgman @theofficialgman ·

        I believe I am also seeing this on ubuntu noble as of about a month ago on a Polaris GPU https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2057155 6.8.0-11-generic #11-Ubuntu SMP PREEMPT_DYNAMIC Wed Feb 14 00:29:05 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux this occurs about 3 times a week for me

        I did not experience this problem on ubuntu mantic (kernel 6.5) or older ubuntu versions

      • Marco Rodolfi
        Marco Rodolfi @RodoMa92 ·

        Nah, seems different to me:

        Mar 11 18:30:07 garrett-desktop kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx timeout, signaled seq=211055, emitted seq=211057
Mar 11 18:30:07 garrett-desktop kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process armcord pid 14844 thread armcord:cs0 pid 14883
Mar 11 18:30:07 garrett-desktop kernel: amdgpu 0000:01:00.0: amdgpu: GPU reset begin!
Mar 11 18:30:07 garrett-desktop kernel: amdgpu 0000:01:00.0: [drm:amdgpu_ring_test_helper [amdgpu]] *ERROR* ring kiq_0.2.1.0 test failed (-110)
Mar 11 18:30:07 garrett-desktop kernel: [drm:gfx_v8_0_hw_fini [amdgpu]] *ERROR* KCQ disable failed

        Different ring, different trigger cause (electron vs idle).

        Feel free to open another bug report in here, tho. Maybe someone will help.

      • Please register or sign in to reply
    • Mario Limonciello
      Mario Limonciello @superm1 ·
      Developer

      Those of you still having this issue on Steam Deck OLED, could you please email your serial number to me? I'd like to share them with the hardware group to see if they're all in the same batch.

      If it's happening on Steam Deck LCD; please make sure your kernel has https://github.com/torvalds/linux/commit/ab4750332dbe535243def5dcebc24ca00c1f98ac applied.

      Edited by Mario Limonciello
    • Natalie Vock mentioned in issue #3440

      mentioned in issue #3440

    • Natalie Vock
      Natalie Vock @pixelcluster ·

      Missed that this issue existed. I reproduced and debugged this issue for a while and made #3440 to share my findings. The current SteamOS Preview includes a workaround for this.

    • Tobias Jakobi
      Tobias Jakobi @tobiasjakobi ·

      Could it be possible that the same problem also affects SDMA v5.0? I'm asking because of this pesky little issue: #1707 (closed)

      Edited by Tobias Jakobi
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    Please register or sign in to reply