[5.2/5.3][drm:amdgpu_dm_atomic_commit_tail [amdgpu]] *ERROR* Waiting for fences timed out or interrupted!

Submitted by udo

Assigned to Default DRI bug account

Link to original bug (#111979)

Description

Seen on both AMD 2400g and 3400g APU's, we find these in dmesg of 5.3.5.:

85.232749] fuse: init (API version 7.31)
[18161.173791] [drm:amdgpu_dm_atomic_commit_tail [amdgpu]] ERROR Waiting for fences timed out or interrupted!
[18166.037697] [drm:amdgpu_job_timedout [amdgpu]] ERROR ring gfx timeout, but soft recovered
[18171.153568] [drm:amdgpu_dm_atomic_commit_tail [amdgpu]] ERROR Waiting for fences timed out or interrupted!
[18186.261621] [drm:amdgpu_job_timedout [amdgpu]] ERROR ring gfx timeout, but soft recovered

or on 5.2.17 sometimes:

[ 7596.392996] sd 11:0:0:0: [sde] Synchronize Cache(10) failed: Result: hostbyte=0x01 driverbyte=0x00
[97954.657336] [drm:amdgpu_dm_atomic_commit_tail [amdgpu]] ERROR Waiting for fences timed out or interrupted!
[97959.535278] [drm:amdgpu_job_timedout [amdgpu]] ERROR ring gfx timeout, signaled seq=2542528, emitted seq=2542531
[97959.535342] [drm:amdgpu_job_timedout [amdgpu]] ERROR Process information: process pid 0 thread pid 0
[97959.535346] [drm] GPU recovery disabled.

Then the graphics stop working and the machine GUI is unusable until reboot.

added AMDgpu bugzilla labels

udo said:

This is on AMD Ryzen 5 3400G with Radeon Vega Graphics, Fedora 30, git mesa, git amdgpu, kernel.org kernel.

udo said:

5.3.6 has similar issues.

5.3.x (13,16) can be quite stable. 5.4.x shows this issue quite quickly though. Much quicker than 5.3. This is with git mesa, kernel.org and git xf86-video-amdgpu.

This means I can easily repdroduce the issue on 5.4; please send patches or settings to find root cause.

It is Fedora 31 in the mean while...

Do you have any specific xorg config? (if yes attach it please) Could you attach xorg.log and a full dmesg log please? Do you have any environment variable that might affect the drivers (e.g: AMD_DEBUG)?

Also: what desktop environment are you using (assuming you're using one)?

xorg.conf:

Section "Monitor"
        Identifier   "Philips"
	DisplaySize 620 343
EndSection

Section "ServerFlags"
	Option "DontZap" "False"
	Option "AIGLX" "True"
EndSection


Section "Device"
        Identifier  "Radeon"
        Driver      "amdgpu"
	Option "DRI3" "on" 
	Option "TearFree" "on"
EndSection

Section "Screen"
        Identifier "MyScreen"
        Device     "Radeon"
        #DefaultDepth     30
        DefaultDepth     24
        SubSection "Display"
                #Depth     30
                Depth     24
                ## This is superfluous and actually harmful with a
                ## static configuration. Enable for dynamic config only.
                #Virtual  1680 1050
        EndSubSection
EndSection

Section "DRI"
        Mode         0666
EndSection

/etc/environment:

# cat /etc/environment
LIBGL_DRIVERS_PATH=/opt/xorg/lib64/dri/
LIBVA_DRIVERS_PATH=/opt/xorg/lib64/dri/
VDPAU_DRIVER_PATH=/opt/xorg/lib64/vdpau/
VDPAU_DRIVER=radeonsi

We run gnome on fedora 31; is that specific enough?

$ rpm -q gnome-shell
gnome-shell-3.34.2-1.fc31.x86_64

dmesg_5.3.16.txt Xorg.0.log

I am not sure if the xorg.log shows the same issue as this bug is about. Next occurrence I will make sure it does.

mentioned in issue #847 (closed)

Be better if you can share step by step instruction how to reproduce this issue.
If you don't know exactly steps, this means this is heisenbug.

5.4: boot into kernel, start thunderbird, firefox, some terminals. Usually crashes within 15 minutes.

5.3.x: takes a bit longer (days...). Simply browse the net, watch a movie, etc.

mesa is configured like this: meson configure . --prefix /opt/xorg/ -Dbuildtype=release -Ddri-drivers=[] -Dgallium-drivers=radeonsi -Dgallium-xvmc=auto -Dgallium-vdpau=true -Dgles1=false -Dgles2=true -Dgallium-xa=auto -Dgallium-opencl=disabled -Ddri3=true -Dplatforms=drm,x11,wayland -Dshared-glapi=true -Dglx=dri -Dglx-direct=true -Dgbm=true -Dosmesa=none -Dglvnd=true -Dlmsensors=true -Dvulkan-drivers=amd -Dgallium-va=true

Did you try to boot without using any amdgpu kernel parameter? Are you using the latest firmwares?

We use:

        Manufacturer: Gigabyte Technology Co., Ltd.
        Product Name: X570 AORUS PRO

Board is at latest version F11 : https://www.gigabyte.com/eu/Motherboard/X570-AORUS-PRO-rev-10/support#support-dl-bios

$ rpm -q linux-firmware
linux-firmware-20191022-103.fc31.noarch

We use amdgpu boot parameters: amdgpu.gttsize=8192 amdgpu.lockup_timeout=0 amdgpu.gpu_recovery=1

First one is for allowing zoom in large PDFs if I recall correctly, the others are related to the bug(s). We did not boot without those yet.

FWIW:

We use a 4K monitor, Gnome scales by factor of 2. We then scale back using xrandr --output HDMI-A-0 --scale 1.25x1.25

Quick crash: dmesg-5.4.1.txt dmesg-5.4.3.txt

As you can see on 5.4.x the issue appears quite quickly. Please explain the root cause if that is known at this time.

@udovdh could you try kernel 5.4.4 ?
Here heisenbug issue was disappeared after kernel upgrade to 5.4.4 (topic starter used internal GPU too)

We have a kernel 5.4.5 ready to go. I will test that one this weekend.

Are we sure this is the issue I was seeing on 5.4.x as it appears so soon?

https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.4.4 does not mention this commit.

Same things still happen.

Screen frezes without any recovery or whatever.

5.4 is unusable this way.

We log in xorg, we start Firefox, terminals, etc and boom!

dmesg-5.4.5.txt

Could you try to boot without any amdgpu kernel parameter and see if you can reproduce the freezes?

without any amdgpu

without any amdgpu kernel parameter GPU reset would happens instead

Without amdpgpu parameters: zonder_amdgpu.txt

Screen froze and sat like that.

If gpu reset would work then that would be a workaround. At the time of the errors we only had a gnome desktop, some terminals, thunderbird and perhaps transmission running. No fancy or weird graphics related software.

The issue of WARNING: CPU: 4 PID: 559 at drivers/gpu/drm/amd/amdgpu/../display/dc/calcs/dcn_calcs.c:1464 dcn_bw_update_from_pplib+0xa5/0x2e0 [amdgpu] (visible in dmesgs) has been open for months...

So what can I try next?

Would 5.4.6 have changes that are relevant to this issue? https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.4.6

I do believe I'm affected by the same issue. For reference this is my bug report for Arch Linux: https://bugs.archlinux.org/task/64866?project=1&string=linux&order=lastedit&sort=desc

Since Linux 5.4 my Ryzen 5 2500U notebook (Vega 8 GPU, HP Envy x360) is completely unstable. To make the notebook usable again I had to go back to Linux-lts 4.19, but there standby is broken, so it is not a good workaround.

Basically I only have to open on Plasma the system settings or Firefox with Webrender enabled. Then the GPU will crash immediately and tries to reset. I do not use any special boot parameter for amdgpu!

Dez 16 22:25:11 pc kernel: [drm:amdgpu_dm_atomic_commit_tail [amdgpu]] *ERROR* Waiting for fences timed out! Dez 16 22:25:11 pc kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx timeout, but soft recovered

I've compiled linux 5.5-rc2 from https://cgit.freedesktop.org/~agd5f/linux/log/?h=drm-next, but unfortunately it didn't help at all.

I can confirm like as said in the archlinux bug report that downgrading to 5.3(.13) kernel doesn't cause hangs anymore (so far, 30 mins in, did webgl benchmarks on Firefox webrender enabled). Using latest 5.4 arch kernel caused hangs just a couple of minutes in using Firefox and seems to be triggered more quickly by opening godot and aseprite.

My hardware is Ryzen 2200G using iGPU only and with temps below 50C when stress testing. Currently on mesa 19.3.1 using RADV.

I could try and help bisect the issue this next weekend.

Please be aware that 5.3.x is not perfect but that 5.4 shows the bug much quicker.

I did boot 5.4.6. As soon as I started Firefox the screen froze. dmesg-5.4.6-ff-start.txt

I've tested Linux 5.5-rc2 from https://cgit.freedesktop.org/~agd5f/linux/log/?h=drm-next, so I couldn't believe Linux 5.4.6 will change anything. But I did the test, broken as before!

And yes, Linux 5.3 worked quite well. Linux >= 5.4 is a really really bad regression! From "works quite OK" to "completely broken".

5.3.x is not "OK"; it still crashes after a few days.

Those issues might be fixed with the fixes in 5.4.6.

I've tested 5.4.6 already, as stated above. It will crash too, there is no difference.

And with "OK" I only meant it doesn't crash after a few seconds, Linux 5.3 worked for me at least for some hours.

Would it help if I posted a diff between the 5.3.18 amdgpu code and the 5.4.6 amdgpu code?

Hi! I can confirm this bug as well. However, i don't get a complete freeze, but the gpu recovers after some seconds (around 10 or so).

[  442.939117] [drm:amdgpu_dm_atomic_commit_tail [amdgpu]] *ERROR* Waiting for fences timed out!
[  447.547798] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx timeout, but soft recovered

Linux x360 5.5.0-0.rc2.git3.1.fc31.x86_64 #1 SMP Sat Dec 21 15:40:33 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Extended renderer info (GLX_MESA_query_renderer):
    Vendor: X.Org (0x1002)
    Device: AMD RAVEN (DRM 3.36.0, 5.5.0-0.rc2.git3.1.fc31.x86_64, LLVM 9.0.0) (0x15dd)
    Version: 19.2.8
    Accelerated: yes
    Video memory: 256MB
    Unified memory: no
    Preferred profile: core (0x1)
    Max core profile version: 4.5
    Max compat profile version: 4.5
    Max GLES1 profile version: 1.1
    Max GLES[23] profile version: 3.2