ring vcn_enc_0.0 timeout cause by linux-firmware 20241210.b00a7f7e

System information

• OS: Arch Linux
• GPU: [AMD/ATI] Navi 22 [Radeon RX 6700/6700 XT/6750 XT / 6800M/6850M XT] [1002:73df] (rev c5)
• Kernel version: 6.12.5-2-cachyos
• Mesa version: Mesa 24.3.1-cachyos1.6
• Desktop manager and compositor: kwin 6.2.4

Describe the issue

When I'm playing games and recording with either OBS or gpu-screen-recorder amdgpu will crash eventually. Without recording everything works just fine.

дек 12 19:45:53 oleg-pc kernel: amdgpu 0000:03:00.0: amdgpu: Dumping IP State
дек 12 19:45:53 oleg-pc kernel: amdgpu 0000:03:00.0: amdgpu: Dumping IP State Completed
дек 12 19:45:53 oleg-pc kernel: amdgpu 0000:03:00.0: amdgpu: ring vcn_enc_0.0 timeout, signaled seq=639945, emitted seq=639946
дек 12 19:45:53 oleg-pc kernel: amdgpu 0000:03:00.0: amdgpu: Process information: process gpu-screen-reco pid 45356 thread gpu-screen:cs0 pid 45373
дек 12 19:45:53 oleg-pc kernel: amdgpu 0000:03:00.0: amdgpu: GPU reset begin!

Regression

Caused by linux-firmware 20241210.b00a7f7e. No such issue on 20241111.b5885ec5

Log files as attachment

crash.log

After extensive testing I found that the culprit was new firmware in linux-firmware 20241210.b00a7f7e package. Reverting to 20241111.b5885ec5 fixes the issue.

Can you please post output of dmesg | grep VCN when booting with the good unaffected fw?

[ 4.183368] [drm] Found VCN firmware Version ENC: 1.32 DEC: 4 VEP: 0 Revision: 0

Can you try if this also happens with this FW https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/amdgpu/navy_flounder_vcn.bin?id=3e78bb66e604d2140ea3628d27024266a181ca14 (from Nov 26).

Move it into /lib/firmware/amdgpu, run sudo mkinitcpio -P and reboot.

I've tested this firmware with recording for 3 hours without any crashes, so I think it's okay. Here's the string from dmesg just to assure that this firmware was indeed loaded: [ 4.110572] [drm] Found VCN firmware Version ENC: 1.33 DEC: 4 VEP: 0 Revision: 3

How long would it usually take you to get the hang? I've been trying to reproduce it myself with gpu-screen-recorder -f 120 -w screen -k h264 -o /tmp/out.mkv and game running, but after 4 hours no issues.

It depends. While playing Factorio it hanged only after 4 hours, but when playing Metaphor: ReFantazio it can crash between 0.5-2 hours.

Update: I've just encountered crash on this new firmware so it is not good. Reverting

дек 19 11:29:41 oleg-pc kernel: [drm] Found VCN firmware Version ENC: 1.33 DEC: 4 VEP: 0 Revision: 3
...
дек 19 13:29:21 oleg-pc kernel: amdgpu 0000:03:00.0: amdgpu:  in process gpu-screen-reco pid 26219 thread gpu-screen:cs0 pid 26240
дек 19 13:29:21 oleg-pc kernel: amdgpu 0000:03:00.0: amdgpu:   in page starting at address 0x000080010129f000 from client 0x12 (VMC)
дек 19 13:29:21 oleg-pc kernel: amdgpu 0000:03:00.0: amdgpu: MMVM_L2_PROTECTION_FAULT_STATUS:0x00000000
дек 19 13:29:21 oleg-pc kernel: amdgpu 0000:03:00.0: amdgpu:          Faulty UTCL2 client ID: unknown (0x0)
дек 19 13:29:21 oleg-pc kernel: amdgpu 0000:03:00.0: amdgpu:          MORE_FAULTS: 0x0
дек 19 13:29:21 oleg-pc kernel: amdgpu 0000:03:00.0: amdgpu:          WALKER_ERROR: 0x0
дек 19 13:29:21 oleg-pc kernel: amdgpu 0000:03:00.0: amdgpu:          PERMISSION_FAULTS: 0x0
дек 19 13:29:21 oleg-pc kernel: amdgpu 0000:03:00.0: amdgpu:          MAPPING_ERROR: 0x0
дек 19 13:29:21 oleg-pc kernel: amdgpu 0000:03:00.0: amdgpu:          RW: 0x0
дек 19 13:29:21 oleg-pc kernel: amdgpu 0000:03:00.0: amdgpu: Dumping IP State
дек 19 13:29:21 oleg-pc kernel: amdgpu 0000:03:00.0: amdgpu: Dumping IP State Completed
дек 19 13:29:21 oleg-pc kernel: amdgpu 0000:03:00.0: amdgpu: ring vcn_enc_0.0 timeout, signaled seq=327546, emitted seq=327547
дек 19 13:29:21 oleg-pc kernel: amdgpu 0000:03:00.0: amdgpu: Process information: process gpu-screen-reco pid 26219 thread gpu-screen:cs0 pid 26240
дек 19 13:29:21 oleg-pc kernel: amdgpu 0000:03:00.0: amdgpu: GPU reset begin!
дек 19 13:29:21 oleg-pc kernel: [drm] Register(0) [mmUVD_POWER_STATUS] failed to reach value 0x00000001 != 0x00000002n
дек 19 13:29:22 oleg-pc kernel: [drm] Register(0) [mmUVD_RB_RPTR] failed to reach value 0x00000080 != 0x00000040n
дек 19 13:29:22 oleg-pc kernel: [drm] Register(0) [mmUVD_POWER_STATUS] failed to reach value 0x00000001 != 0x00000002n
дек 19 13:29:22 oleg-pc kernel: DMAR: DRHD: handling fault status reg 3
дек 19 13:29:22 oleg-pc kernel: DMAR: [DMA Write NO_PASID] Request device [03:00.0] fault addr 0x105296000 [fault reason 0x05] PTE Write access is not set

changed title from ring vcn_enc_0.0 timeout when playing games and recording with hardware encoder to ring vcn_enc_0.0 timeout cause by linux-firmware 20241210.b00a7f7e

changed the description

moved from mesa/mesa#12318 (moved)

added 6000 dGPU Series VCN labels

marked #3822 (closed) as a duplicate of this issue

marked this issue as related to #3822 (closed)

After a recent Gentoo upgrade (starting with mesa) I experienced crashes via Chromium. My crashes can be described a freezing screen after which kworker/u32:0+amdgpu-reset-dev causes an extremely high load. Typically from having a YouTube video running or working with SketchUp Web. I have already reverted the firmware back to the release as of 20241110, but the device still crashes.

If this issue is totally unrelated, please set me know.

[ 8332.887809] amdgpu 0000:05:00.0: amdgpu: failed to write reg 28b4 wait reg 28c6
[ 8353.217798] amdgpu 0000:05:00.0: amdgpu: failed to write reg 1a6f4 wait reg 1a706
[ 8353.881008] amdgpu 0000:05:00.0: amdgpu: Dumping IP State
[ 8368.900873] hrtimer: interrupt took 394179891 ns
[ 8476.863375] INFO: task chrome:disk$0:1047 blocked for more than 122 seconds.
[ 8476.863384]       Tainted: P           O       6.12.7-gentoo #1
[ 8476.863387] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 8476.863388] task:chrome:disk$0   state:D stack:0     pid:1047  tgid:992   ppid:964    flags:0x00004002
[ 8476.863395] Call Trace:
[ 8476.863398]  <TASK>
[ 8476.863401]  __schedule+0x3e4/0xa40
[ 8476.863410]  ? xas_load+0x8/0xc0
[ 8476.863415]  schedule+0x26/0xf0
[ 8476.863419]  schedule_timeout+0x121/0x150
[ 8476.863423]  ? trace_hardirqs_on+0x1d/0x80
[ 8476.863428]  ? preempt_count_add+0x69/0xa0
[ 8476.863432]  dma_fence_default_wait+0x17d/0x1f0
[ 8476.863438]  ? dma_fence_signal+0x50/0x50
[ 8476.863443]  dma_fence_wait_timeout+0x104/0x130
[ 8476.863449]  amdgpu_vm_fini+0x11a/0x4f0 [amdgpu]
[ 8476.863569]  ? idr_destroy+0x72/0xb0
[ 8476.863572]  amdgpu_driver_postclose_kms+0x184/0x260 [amdgpu]
[ 8476.863673]  drm_file_free+0x21e/0x270 [drm] 
[ 8476.863689]  drm_release+0xa6/0x100 [drm]
[ 8476.863704]  __fput+0xd6/0x290
[ 8476.863710]  task_work_run+0x55/0x90
[ 8476.863714]  do_exit+0x2c3/0x9e0
[ 8476.863717]  ? __futex_wake_mark+0x50/0x50
[ 8476.863722]  do_group_exit+0x2c/0x80
[ 8476.863724]  get_signal+0x83f/0x8d0
[ 8476.863728]  arch_do_signal_or_restart+0x2a/0x200
[ 8476.863732]  ? do_futex+0xc7/0x190
[ 8476.863735]  ? __x64_sys_futex+0x108/0x1c0
[ 8476.863737]  ? _raw_spin_unlock_irq+0x18/0x40  
[ 8476.863740]  ? get_signal+0x762/0x8d0
[ 8476.863743]  syscall_exit_to_user_mode+0xde/0x150    
[ 8476.863747]  do_syscall_64+0x85/0x120
[ 8476.863749]  ? xfd_validate_state+0x1e/0x80
[ 8476.863752]  ? restore_fpregs_from_fpstate+0x38/0xc0
[ 8476.863756]  ? switch_fpu_return+0x55/0xd0
[ 8476.863759]  ? trace_hardirqs_on_prepare+0x1d/0x80
[ 8476.863762]  ? syscall_exit_to_user_mode+0x6e/0x150
[ 8476.863765]  ? do_syscall_64+0x85/0x120
[ 8476.863767]  ? sched_setaffinity+0x17b/0x210
[ 8476.863771]  ? fpregs_assert_state_consistent+0x21/0x50
[ 8476.863775]  ? trace_hardirqs_on_prepare+0x1d/0x80
[ 8476.863777]  ? syscall_exit_to_user_mode+0x6e/0x150
[ 8476.863780]  ? do_syscall_64+0x85/0x120
[ 8476.863782]  ? trace_hardirqs_on_prepare+0x1d/0x80
[ 8476.863785]  entry_SYSCALL_64_after_hwframe+0x55/0x5d
[ 8476.863789] RIP: 0033:0x7fa22f5540e5
[ 8476.863791] RSP: 002b:00007fa2134fb8d0 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[ 8476.863794] RAX: fffffffffffffe00 RBX: 0000000000000000 RCX: 00007fa22f5540e5
[ 8476.863797] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 00000bac00d9c0a8
[ 8476.863798] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[ 8476.863800] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[ 8476.863801] R13: 0000000000000000 R14: 00000bac00d9c050 R15: 00000bac00d9c0a8
[ 8476.863804]  </TASK>

Issue is reported https://bugs.gentoo.org/947181 since I am running a Gentoo kernel (my own config) at this point running the latest mesa (25) with llvm 19.

That looks unrelated, please open new issue.

Done: mesa/mesa#12407 (moved)

mentioned in issue #3855 (closed)

Would also like to point out this happens with the wivrn-server VR streaming software as well on a 6600 XT.

Are you on ArchLinux too? Have you tried if downgrading linux-firmware to 20241111.b5885ec5 helps?

I was on Fedora 41. Had these issues after switching away from openSUSE. Installed arch and also kept having the issues there, downgrading firmware did fix it there. However downgrading it on Fedora didn't fix it for me.

mentioned in issue mesa/mesa#12420 (moved)

marked this issue as related to #3880 (closed)

marked this issue as related to #3855 (closed)

marked #3880 (closed) as a duplicate of this issue

marked #3855 (closed) as a duplicate of this issue

navy_flounder_vcn.bin We found this is a FW issue, and please use this attached FW to try.

Please test it if possible, so that we can safely update to this FW version. I have tested locally and working fine.

Just adding my experience in here but i havent had this occur on my 680m in my 7735hs with VCN firmware Version ENC: 1.33 DEC: 4 VEP: 0 Revision: 3. The firmware being reverted upstream absolutely destroyed encode quality when streaming on twitch as i previously made an issue about on the mesa tracker. mesa/mesa#12108 (closed).

Luckily i still have the other firmware and can just use it but hopefully this can be resolved in a better way that doesnt leave the 680m encoder in a sorry state. Is there some difference in the 7735hs that might account for me not running into this? or perhaps its due to using 720p exclusively for my encodes?

It will be fixed in new firmware version, the revert is just temporary solution.

@nowrep Has the new firmware landed to upstream?

Not yet.

Can we expect fixes on the linux-firmware tree soon? I'm using Navy Flounder, and there is still a version mismatch between VCN and driver v6.3.2 (AMF 1.4.36). I have been testing the custom version Ruijing posted as a workaround for now, and everything seems to work fine.

There seem to be some VCN firmware updates pushed to linux-firmware: https://web.git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/

Yes, these have fix for this issue.

This didnt just fix the issue but crushed it and others into oblivion. The quality of the my streams is now the best its even been on an AMD GPU. Genuinely surprised and it was such a huge difference as to have people comment on it without me making any settings changes in OBS.

I would argue that this shows the importance of having open source firmwares as well. Imagine engineers really go for a specific task that they care about, for example video encoding. If AMD can keep up with their press releases I hope this happens in 2026.

closed