Regression - Fence fallback timer expired on ring - Virtualized environment

Brief summary of the problem:

Context: I am using QubesOS (based on the xen hypervisor) I am using my GPU in a VM. To do that I am doing a GPU passthrough (AMD RX 580). The VM is a linux HVM (archlinux). I was using kernel 5.4 LTS (the linux kernel of the vm), and everything was working fine, and everything work fine up to kernel 5.6 included.

I upgraded the kernel to a more recent version and then the GPU was unusuably slow, with many error showing up in the kernel log. The issue can always be reproduced. Start the VM. Error will always show up in the kernel log. If you try to use the GPU (like starting xorg), everything will be unusuably slow.

[    8.051268] [drm] Fence fallback timer expired on ring gfx
[    8.561265] [drm] Fence fallback timer expired on ring comp_1.0.0
[    9.071264] [drm] Fence fallback timer expired on ring comp_1.1.0
[    9.581239] [drm] Fence fallback timer expired on ring comp_1.2.0
[   10.091281] [drm] Fence fallback timer expired on ring comp_1.3.0
[   10.601270] [drm] Fence fallback timer expired on ring comp_1.0.1
[   11.111265] [drm] Fence fallback timer expired on ring comp_1.1.1
[   11.621354] [drm] Fence fallback timer expired on ring comp_1.2.1
[   12.131265] [drm] Fence fallback timer expired on ring comp_1.3.1
[   12.641289] [drm] Fence fallback timer expired on ring sdma0
[   13.151264] [drm] Fence fallback timer expired on ring sdma1
[   13.681279] [drm] Fence fallback timer expired on ring uvd
[   14.191309] [drm] Fence fallback timer expired on ring uvd_enc0
[   14.701269] [drm] Fence fallback timer expired on ring uvd_enc1
[   15.311266] [drm] Fence fallback timer expired on ring vce0
[   15.821289] [drm] Fence fallback timer expired on ring vce0
[   93.821281] [drm] Fence fallback timer expired on ring sdma1
[   94.471284] [drm] Fence fallback timer expired on ring sdma1
[   95.531275] [drm] Fence fallback timer expired on ring sdma1
[   96.181291] [drm] Fence fallback timer expired on ring sdma0

I bisected the issue and sent the details here #1381 (comment 1700698)

By applying the modification If I revert back the integer comparison from amdgpu_runtime_pm != 0 back to amdgpu_runtime_pm > 0, then this issue disappear. it fix this issue, from kernel 5.7 to 5.9 included. Something new have been added/modified between 5.9 and 5.10 and break things again. I am in the process of bisecting the second issue/regression, but it is going to take few days

Hardware description and system information:

• CPU: Ryzen 7 1700
• GPU: AMD RX 580
• System Memory: 32GB
• Type of Display Connection: HDMI

How to reproduce the issue:

• Install QubesOS (it is based on the xen hypervisor, i do not think the "qubes" part have any link to the issue)
• Configure the GPU passthrough, using a archlinux HVM (or any other linux system, tested the same issue with a debian previously) https://neowutran.ovh/qubes/articles/gaming_windows_hvm.html https://neowutran.ovh/qubes/articles/gaming_linux_hvm.html
• Start the VM
• Look at the kernel log sudo dmesg
• Try to start xorg or anything that use the GPU

You can use amdgpu.runpm=0 to disable runtime power management when you load the driver. It sounds like either runtime power management or MSIs are not working in the guest VM. Runtime power management relies on the OS putting the device into D3 at runtime. If the guest is not doing that the the driver portion of that won't work either.

Thanks you a lot ! :)

It seems that setting amdgpu.runpm=0 and also setting pci=nomsi fix the issue (to be confirmed, going to take few days, but from my first test it seems to do the trick)

So the problem (if confirmed) is probably on the xen side. I do not really known what "pci=nomsi" does, do you known what it means to have to set those 2 parameter ?

Does just setting pci=nomsi alone fix it or do you need both parameters?

MSIs (https://en.wikipedia.org/wiki/Message_Signaled_Interrupts) are a way of delivering interrupts on modern devices. It would appear xen does not handle them correctly.

For runtime power management, the OS generally puts the device into D3 state when it's idle. The D state is controlled via PCI config space which the hypervisor generally owns. When the guest requests a change to PCI config space it's generally either proxied to the hypervisor or dropped. I'm not sure how xen handles this. If it's dropped, it won't work.

setting pci=nomsi alone fix it, no need for amdgpu.runpm=0

And thanks for the explication

So I guess it means that MSI are dropped or ignored or not handled correctly. But theoretically it should not be the case ( Opened a ticket in the QubesOS issue tracker https://github.com/QubesOS/qubes-issues/issues/7971 )

Yeah, it should be possible to use MSIs in VMs.

Your description in https://github.com/QubesOS/qubes-issues/issues/7971 is not quite correct. MSIs have nothing to do with power management and amdgpu has been using MSIs since the driver was created. MSIs are used for interrupts.

See amdgpu_irq_init() in amdgpu_irq.c for how the driver interacts with the pci subsystem for allocating interrupts. The driver does use MSI-X if it's available.

Thanks, will check that code, and report the details to the qubes project. Since it was working before linux kernel 5.7, does this means that only some MSI call are not working ? And that the unsupported MSI call have been introduced in 5.7 and 5.10 ?

At some point the driver switched from using MSIs only to using MSI-X if available. That might have been in that timeframe.

MSI/MSI-X situation in qubes currently is weird. MSI should work just fine. But MSI-X does not yet, and is hidden away from PCIe caps by device model. So, if driver is looking for MSI, it's fine. If it expects MSI-X, it won't find it. But then, it may either fallback to MSI (which generally works fine), or INTx (which we find broken in several drivers, as that path is rarely tested for MSI-supporting devices).

Proper MSI-X support is not far away, but we still have some work to do in that area.

Is there a way to tell the driver to use MSI in absence of MSI-X? pci=nomsi is a global switch that disables both...

Shouldn't pci_alloc_irq_vectors() fallback to MSIs if MSI-X is not available? I suppose the driver could check if we are in a xen vm and not set the PCI_IRQ_MSIX flag, but that seems like a hack. The PCI core should handle this transparent to the driver.

I see currently both PCI_IRQ_MSIX and PCI_IRQ_MSI flags are set, so theoretically it should be okay. Skipping PCI_IRQ_MSIX on Xen in upstream driver indeed sounds bad (the issue with MSI-X is specific to our configuration - namely using qemu in stubdomain; it should work just fine in default Xen setup). I may include such patch in our kernel, until proper MSI-X support is done, though. But I'd need to double check if that really helps first.

Thanks for the help!

closed

It strange but I have exactly the same error on host OS without virtual machines.