[RX 5600XT] Random GPU hang while playing games, GPU reset

problem_n10_kernel6.txt problem_n11_kernel6.txt

After 6 days of error-free gaming, unfortunately I got this problem with Kernel 6.0 too. The log is a bit different than the older logs.

I think that there is a sign of the problem, visible tearing in the browser while scrolling. But it could also be a coincidence.

*edit: I got a second GPU reset today... :(

Confirm having this problem too, wasn't sure if it was my desktop manager or gpu related but going to go with the safe bet it's gpu, however I have not had this problem until upgrading past 5.19.11 just recently to 6.0.2 due to #2189 (closed)

What might be interesting to note is we both use KDE. It is possible it is a KDE 5.26 bug, but since you've been having this for a while I suspect that's not the case

At the moment, I have KDE 5.25, but yeah, I have this issue more or less constantly since months.

Now I'm switched back to the 5.15 LTS kernel. Will see if that fix this issue or not.

problem_n12_kernel5_15.txt

The problem is present with kernel 5.15 LTS too. :( See the attached dmesg output.

Now I'm trying kernel 5.10 LTS. Sigh.

problem_n13_kernel5_10.txt

Well, this was fast. I got a crash after 1 minute of gameplay. Then the GPU reset actually triggered a complete PC restart (yes, this was usual for me at the time of this old kernel).

Now I'm moving to kernel 6.1RC. That may work better. - or not. Gaming does not work on 6.1RC at all. So it is 6.0.2 then. I have no better idea.

Yeah, with kernel 6.0.6, the problem are still here...

nov 04 22:33:09 kepsz-ga78lmts2p kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx_0.0.0 timeout, signaled seq=1270116, emitted seq=1270118
nov 04 22:33:09 kepsz-ga78lmts2p kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process stellaris pid 12173 thread stellaris:cs0 pid 12178
nov 04 22:33:13 kepsz-ga78lmts2p kernel: amdgpu 0000:08:00.0: [drm:amdgpu_ring_test_helper [amdgpu]] *ERROR* ring kiq_2.1.0 test failed (-110)
nov 04 22:33:13 kepsz-ga78lmts2p kernel: [drm:gfx_v10_0_hw_fini [amdgpu]] *ERROR* KGQ disable failed
nov 04 22:33:14 kepsz-ga78lmts2p kernel: [drm:gfx_v10_0_hw_fini [amdgpu]] *ERROR* failed to halt cp gfx
nov 04 22:33:16 kepsz-ga78lmts2p kernel: [drm:amdgpu_cs_ioctl [amdgpu]] *ERROR* Failed to initialize parser -125!
nov 04 22:33:16 kepsz-ga78lmts2p kernel: [drm:amdgpu_cs_ioctl [amdgpu]] *ERROR* Failed to initialize parser -125!
nov 04 22:33:16 kepsz-ga78lmts2p kernel: [drm:amdgpu_cs_ioctl [amdgpu]] *ERROR* Failed to initialize parser -125!
nov 04 22:33:16 kepsz-ga78lmts2p kernel: [drm:amdgpu_cs_ioctl [amdgpu]] *ERROR* Failed to initialize parser -125!
nov 04 22:33:16 kepsz-ga78lmts2p kernel: [drm:amdgpu_cs_ioctl [amdgpu]] *ERROR* Failed to initialize parser -125!

problem_n15_kernel606.txt

Finally we will have a solution? https://www.phoronix.com/news/AMDGPU-Fix-For-5.19-Bug

Can you confirm that the patch(es) linked in that article fix this issue for you?

I don't know how to test it. At the moment, I can install kernel 6.1 RC04 from Manjaro, but that's the latest kernel in our repos. Afaik there are a kernel source git repo for the latest graphics patches, but I'm not sure. It's a bit hard to find info about this things.

If you are not comfortable compiling the kernel on your own, better wait until the patches go downstream. That may take time though. Unfortunately the kernel maintainers have a very clumsy development cycle.

I did compiled various kernels, even for embedded systems, it's not a problem. I just don't know where it is located. Google was not helpful

I am not familiar with Manjaro (or Arch) but there are various results on the Arch wiki and Manjaro forums on how to compile your custom kernel. On Fedora which I use, all I need is: first clone the kernel repo, check out the latest RC then apply any patches I need, then:

# Configure the kernel according to the currently running distro kernel
make olddefconfig
# Compile everything on 16 threads
make -j16
# Install all modules and the bootloader
sudo make modules_install install

I am not sure if the same will work on Manjaro or not, please seek advice from the Manjaro community.

Now I get it. I have to apply that patch manually to Manjaro's kernel source. I will try to do it.

Ok, I was able to compile the 6.0.7 kernel, with 2 files modified according this commit: agd5f/linux@8a1a7d74

Afaik that commit is the patch for the "timed out fences" problem. It made no difference, I got the same problem running gputest: problem_tamaskernel_2.txt

mentioned in issue mesa/mesa#7664 (closed)

Quite easy, just play some games.

Unfortunately this is not really a useful list of steps to reproduce. I used to play some games on the same GPU (Navi 10 - RX 5700XT) which didn't exhibit this problem. From what you described thus far, this could either be a kernel issue (related to power management or whatever else), or a userspace driver issue with the specific game you are playing.

Some of your logs contain:

[ 1547.632494] [drm:amdgpu_job_timedout [amdgpu]] ERROR ring gfx_0.0.0 timeout, signaled seq=336297, emitted seq=336299

and another one has:

okt 10 19:19:52 kepsz-ga78lmts2p kernel: [drm:amdgpu_job_timedout [amdgpu]] ERROR ring comp_1.2.0 timeout, signaled seq=11613, emitted seq=11614

It seems that on one occasion the hang occoured on the GFX ring (graphics queue) and on another occasion the compute queue. Judging by how unstable the machine is, I recommend to first try to rule out power management related issues.

Games: Cyberpunk, Stellaris, Star Trek Online, World War Z.

"Judging by how unstable the machine is, I recommend to first try to rule out power management related issues." - what do you recommend? I did lengthy stress tests under both Linux and Windows, but the machine(s) remained rock solid. I even tested the situation where both CPU and GPU was loaded to maximum.

What kind of stress test did you do?

On Linux: -Unigine Superposition benchmark

On Windows: -Unigine Superposition benchmark

-Rthdribl

-Furmark

-Furmark + Intel Burn Test for maximum power drawn

Okay, so on Linux you ran Unigine Superposition and on Windows you ran the same and various others, right? Making sure things are stable on Windows is useful because it rules out any hardware defect. Was there anything else you tried to run on Linux? And how long did you run Unigine Superposition for?

Unigine was run for just some test runs. But now I was able to reproduce this issue with gputest on Linux, just under 5 minutes.

Dmesg: problem_n16_kernel606.txt

This last log contains a kernel crash (unlike the previous logs). Can you elaborate on what exactly you did? Did you use this tool?

Yes, that tool. I just let gputest 0.7.0 (with the default tricolour triangle) run while I used Firefox to read the news. Why do you think it was a kernel crash? I see the same / similar issue there:

[ 7528.559420] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx_0.0.0 timeout, signaled seq=21355393, emitted seq=21355395
[ 7528.559655] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process Xorg pid 1016 thread Xorg:cs0 pid 1220
[ 7528.559842] amdgpu 0000:08:00.0: amdgpu: GPU reset begin!
[ 7532.559894] amdgpu 0000:08:00.0: amdgpu: failed to suspend display audio
[ 7532.882661] amdgpu 0000:08:00.0: [drm:amdgpu_ring_test_helper [amdgpu]] *ERROR* ring kiq_2.1.0 test failed (-110)
[ 7532.882777] [drm:gfx_v10_0_hw_fini [amdgpu]] *ERROR* KGQ disable failed
[ 7533.119190] [drm:gfx_v10_0_hw_fini [amdgpu]] *ERROR* failed to halt cp gfx
[ 7533.134841] [drm] free PSP TMR buffer

Meanwhile, I bought the paid version of Unigine Superposition, and with it I was able to run it's stress test for more than an hour. I got no crash so far.

I think it's a crash because it has a backtrace:

[ 7535.490405] Call Trace:
[ 7535.490406]  <TASK>
[ 7535.490407]  drm_sched_main+0x63/0x440 [gpu_sched 9c2863ba76128d465cc2f68df6c14c5b94dadfd2]
[ 7535.490413]  ? cpuacct_percpu_seq_show+0x20/0x20
[ 7535.490416]  ? drm_sched_resubmit_jobs+0x20/0x20 [gpu_sched 9c2863ba76128d465cc2f68df6c14c5b94dadfd2]
[ 7535.490419]  kthread+0xde/0x110
[ 7535.490421]  ? kthread_complete_and_exit+0x20/0x20
[ 7535.490423]  ret_from_fork+0x22/0x30
[ 7535.490427]  </TASK>
[ 7535.490427] ---[ end trace 0000000000000000 ]---

But you are right, the GPU hang seems to happen even before that:

[ 7528.559420] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx_0.0.0 timeout, signaled seq=21355393, emitted seq=21355395

It's not a crash just a warning. I think this patch series should fix the warning: https://patchwork.freedesktop.org/series/110182/

@kepszlok So, gputest reproduces the hang under 5 minutes, but Superposition does not?

@Venemo Jeah, but this is quite random. Right now gputest is running fine since about half an hour... Some times I can play for hours and everything is OK, but other times it just crash when it wants.

Okay. I do recommend to try to rule out power management related problems. Disable any overclocking you may have on your system and set the GPU to one of the fixed-clock profiles. Then see if that improves it or not. That is unless @agd5f has a better suggestion.

To fix the GPU clock profiles... Should I set "Performance mode" from Automatic to Fixed/High in CoreCtrl tool?

Well, I was able to cause this crash 5 times in a row using gputest tool. It happens usually under 5 minutes of test run. Gputest tool also caused this crash at my other PC (with same mainboard, cpu, vga and OS). So this tool is good to reproduce the problem.

gputest tool was also able to cause this crash under Linux Mint, with kernel 5.15.

problem_lmint_k515.txt

changed the description

Chiming in, I had these hangs too on kernel 6, random crashes, not related to load whatshowever. 5.15lts fixed it for me

I was able to install drm-next-tip kernel from AUR. That too has this problem.

Next thing I will try is to investigate this problem using live system from pendrive.

Well, that didn't went fine. I did tested these live iso's:

Manjaro KDE

Regata OS

Pop OS!

SteamOS

All of them produced the same GPU error in under 5 minutes by running GpuTest tool.

Woah, the default "triangles" test of GpuTest is broken. I tried it on Windows too and that test crashed the gpu driver under 2 minutes. But crashed it so hard, I had to reinstall the whole driver o.O

Back to Linux... Now I'm running the Furmark test from GpuTool. That did not crashed the driver so far.

changed the description

In these days, Star Trek Online is barely playable, because the driver dies in under 5 minutes. New type of errors:

[  461.721328] [drm:gfx_v10_0_priv_reg_irq [amdgpu]] *ERROR* Illegal register access in command stream
[  461.731467] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx_0.0.0 timeout, signaled seq=126716, emitted seq=126718
[  461.731591] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process Main Thread pid 3119 thread dxvk-submit pid 3145
[  461.731702] amdgpu 0000:09:00.0: amdgpu: GPU reset begin!

[  461.721328] [drm:gfx_v10_0_priv_reg_irq [amdgpu]] *ERROR* Illegal register access in command stream

This is a new one. Would be interesting to find out which register is being illegally accessed there.

Do you get this message in Star Trek Online only?

For now, only with STO. But I had very little time to play games in these days.

Is there a way to get some more helpful debug info out from amdgpu / dmesg?

You can look into umr, but it may be complicated.

Stellaris did crashed in the "usual" way, without the illegal register thing:

[ 3764.769063] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx_0.0.0 timeout, signaled seq=516115, emitted seq=516117
[ 3764.769279] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process stellaris pid 5362 thread stellaris:cs0 pid 5367
[ 3764.769466] amdgpu 0000:09:00.0: amdgpu: GPU reset begin!
[ 3768.769490] amdgpu 0000:09:00.0: amdgpu: failed to suspend display audio
[ 3769.116068] amdgpu 0000:09:00.0: [drm:amdgpu_ring_test_helper [amdgpu]] *ERROR* ring kiq_2.1.0 test failed (-110)
[ 3769.116182] [drm:gfx_v10_0_hw_fini [amdgpu]] *ERROR* KGQ disable failed
[ 3769.369121] [drm:gfx_v10_0_hw_fini [amdgpu]] *ERROR* failed to halt cp gfx
[ 3769.384853] [drm] free PSP TMR buffer
[ 3769.418457] amdgpu 0000:09:00.0: amdgpu: BACO reset
[ 3771.538431] amdgpu 0000:09:00.0: amdgpu: GPU reset succeeded, trying to resume
[ 3771.538570] [drm] PCIE GART of 512M enabled (table at 0x0000008000F00000).

Stellaris is a native Linux game, but STO runs with Proton (Proton-experimental in my case). Proton could be the source of the illegal register problem... I will try to test this with Cyberpunk. And will check out umr too.

Proton does not access any registers on your GPU. The driver does.

Hmm yeah.

But I did discovered something today playing with CoreCtrl and vga settings. There was a couple of guys that had similar ring timeout problems and some of them fixed it by running the card in the fixed low power mode. https://bugzilla.kernel.org/show_bug.cgi?id=201957

By default, the card can draw 135W power, with 1780MHz core and 1500MHz ram clocks. In reality, the core clock is around 1550Mhz while the card is stressed with furmark. With these default settings, STO does crash just under minutes.

But with power limit set to 115W, I was able to play STO for more than 2 hours, it was crash-free. With this setting, the core clock is around 1400MHz at full load, that means I still get around 90% of the card's maximum performance. I can live with this if it fixes the problem in return... This mode will need some more testing.

I still think this is a pure software issue, because of this card runs fine under Windows at the default settings, with 135W power draw. There are no overheat or power supply related problems.

Great, thanks for the testing. Please include some information here about what exactly you did. If your GPU works correctly with those settings, then this is 99% a power management issue, likely a bug in the kernel driver.

I'm using these settings:

Btw, testing this will take time. For example, STO with 125W power limit does still crashed, but not that soon like with the default settings. This is why I'm testing 115W power limit for now.

If power management settings affect the stability of a game this seriously, I'm pretty sure this is going to be a power management issue.

Can you say exactly which 5600XT model you have? Ie. which manufacturer made it etc.?

Yes, in both of my PC's, I have this card: Sapphire Pulse 5600XT - this is the "normal" version, they have a "Black edition" variant, but that's just a cheep junk that loud as a hair dryer.

Graphics:
  Device-1: AMD Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT]
    vendor: Sapphire driver: amdgpu v: kernel arch: RDNA-1 pcie: speed: 16 GT/s
    lanes: 16 ports: active: HDMI-A-1 empty: DP-1,DP-2,DP-3 bus-ID: 09:00.0
    chip-ID: 1002:731f class-ID: 0300

I did experienced one crash with Stellaris using 115W as power limit... Now I'm testing 100W.

I have a suspicion that something goes wrong when booting... In many cases, such a crash is preceded by some video stuttering, but not so much that you have to restart the machine because of it. It is difficult to decide whether there is a driver reason for this, or e.g. the game itself is problematic. And it doesn't even show up before every crash. So this is just a suspicion on my part.

I did some cross-checking and may have found the reason for all of these errors.

I ran GpuTest with furmark stress test on both Linux and Windows. The GPU voltages should be similar under load, but they didn't.

Under Windows, the GPU voltage stayed between 900 - 850mV range but under Linux, it started from 850mV and goes even below 834mV. - at same clock speeds.

Until now, I thought that the frequency-voltage curve settings were the same under all operating systems, because they are defined in the VGA bios - at least until someone changes it in the driver. GPUs are pushed to the limit, if they don't get the required voltage, it can easily cause freezing.

Can you say how you checked the voltages?

With Furmark is running,

On Windows, I watched it in Amd software, under the tuning (manual) tab.

On Linux, I watched it in CoreCtrl.

Does CoreCtrl allow you to manually adjust the voltage? If you change it to match what you see in Windows does that help make it more stable?

No, it can not change the voltage. I will check other tools/methods for that.

mentioned in issue mesa/mesa#7741 (closed)

[RX 5600XT] Random GPU hang while playing games, GPU reset

Brief summary of the problem:

Hardware description:

System information:

How to reproduce the issue:

Attached files:

Designs

Child items ...

Activity

Admin message

Admin message

[RX 5600XT] Random GPU hang while playing games, GPU reset

Brief summary of the problem:

Hardware description:

System information:

How to reproduce the issue:

Attached files:

Activity