WARNING in kernel/workqueue.c:3094 __flush_work.isra.0+0x20e/0x220 when accessing /dev/kfd

Same

Ryzen 1800x / Radeon VII
Linux debamd 5.15.0-3-amd64 #1 (closed) SMP Debian 5.15.15-2 (2022-01-30) x86_64 GNU/Linux
debian:testing

[99971.767877] ------------[ cut here ]------------
[99971.767880] WARNING: CPU: 14 PID: 183669 at kernel/workqueue.c:3083 __flush_work.isra.0+0x21b/0x230
[99971.767889] Modules linked in: xt_nat xt_tcpudp veth xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo nft_counter xt_addrtype nft_compat nf_tables libcrc32c nfnetlink br_netfilter bridge stp llc snd_seq_dummy snd_hrtimer snd_seq snd_seq_device rfkill overlay binfmt_misc intel_rapl_msr intel_rapl_common edac_mce_amd kvm_amd nls_ascii ppdev kvm snd_hda_codec_realtek nls_cp437 vfat snd_hda_codec_generic fat irqbypass ledtrig_audio snd_hda_codec_hdmi ghash_clmulni_intel snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec snd_hda_core aesni_intel snd_hwdep snd_pcm crypto_simd cryptd rapl snd_timer joydev snd ccp serio_raw efi_pstore wmi_bmof soundcore pcspkr rng_core sg sp5100_tco watchdog k10temp parport_pc parport evdev acpi_cpufreq msr fuse configfs efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic hid_generic usbhid hid sd_mod amdgpu nouveau mxm_wmi gpu_sched video
[99971.767955]  i2c_algo_bit drm_ttm_helper ttm xhci_pci ahci drm_kms_helper xhci_hcd libahci libata cec rc_core usbcore nvme drm psmouse r8169 scsi_mod nvme_core crc32_pclmul crc32c_intel realtek t10_pi crc_t10dif crct10dif_generic mdio_devres gpio_amdpt crct10dif_pclmul i2c_piix4 libphy scsi_common usb_common crct10dif_common wmi gpio_generic button
[99971.767980] CPU: 14 PID: 183669 Comm: rocminfo Tainted: G        W         5.15.0-3-amd64 #1  Debian 5.15.15-2
[99971.767984] Hardware name: Micro-Star International Co., Ltd. MS-7B89/B450M MORTAR (MS-7B89), BIOS 1.00 07/16/2018
[99971.767986] RIP: 0010:__flush_work.isra.0+0x21b/0x230
[99971.767990] Code: 8b 4d 00 4c 8b 45 08 89 ca 48 c1 e9 04 83 e2 08 83 e1 0f 83 ca 02 89 c8 48 0f ba 6d 00 03 e9 17 ff ff ff 0f 0b e9 49 ff ff ff <0f> 0b 45 31 ed e9 3f ff ff ff e8 e6 8f 86 00 66 0f 1f 44 00 00 0f
[99971.767992] RSP: 0018:ffffbe3d4af27c60 EFLAGS: 00010246
[99971.767996] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[99971.767998] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9e319826ab18
[99971.767999] RBP: ffff9e319826ab18 R08: 0000000000000000 R09: 0000000000000000
[99971.768001] R10: 0000000000000000 R11: 000000000000000e R12: ffff9e319826ab18
[99971.768002] R13: 0000000000000001 R14: 0000000000000002 R15: ffff9e362359d5ac
[99971.768004] FS:  00007f63dbac4780(0000) GS:ffff9e389ef80000(0000) knlGS:0000000000000000
[99971.768007] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[99971.768008] CR2: 00007f63dbadcabc CR3: 0000000202e60000 CR4: 00000000003506e0
[99971.768011] Call Trace:
[99971.768013]  <TASK>
[99971.768015]  ? __wait_for_common+0x3b/0x160
[99971.768020]  ? srcu_gp_start_if_needed+0x237/0x3e0
[99971.768024]  __cancel_work_timer+0x112/0x1a0
[99971.768028]  kfd_process_notifier_release+0x8b/0x150 [amdgpu]
[99971.768278]  __mmu_notifier_release+0x73/0x210
[99971.768283]  exit_mmap+0x1a9/0x1f0
[99971.768287]  ? delayacct_add_tsk+0x63/0x1b0
[99971.768291]  ? exit_robust_list+0x5c/0x140
[99971.768294]  ? __cond_resched+0x16/0x40
[99971.768296]  ? mutex_lock+0xe/0x30
[99971.768298]  mmput+0x56/0x140
[99971.768301]  do_exit+0x2f4/0xa80
[99971.768305]  do_group_exit+0x33/0xa0
[99971.768308]  __x64_sys_exit_group+0x14/0x20
[99971.768312]  do_syscall_64+0x3b/0xc0
[99971.768315]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[99971.768318] RIP: 0033:0x7f63dbd9f6b9
[99971.768321] Code: Unable to access opcode bytes at RIP 0x7f63dbd9f68f.
[99971.768322] RSP: 002b:00007ffceeef1938 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
[99971.768324] RAX: ffffffffffffffda RBX: 00007f63dbe93610 RCX: 00007f63dbd9f6b9
[99971.768326] RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000
[99971.768327] RBP: 0000000000000000 R08: ffffffffffffff50 R09: 0000000000000000
[99971.768328] R10: 00007f63dbee4be1 R11: 0000000000000246 R12: 00007f63dbe93610
[99971.768330] R13: 000000000000003f R14: 00007f63dbe93ae8 R15: 0000000000000000
[99971.768332]  </TASK>
[99971.768333] ---[ end trace 04e2133dfdc6c7a7 ]---
[103863.630873] br-90d91c6c7798: port 1(veth0f28c38) entered disabled state
[103863.630939] veth0cd3b39: renamed from eth0
[103863.707665] br-90d91c6c7798: port 1(veth0f28c38) entered disabled state
[103863.708373] device veth0f28c38 left promiscuous mode
[103863.708377] br-90d91c6c7798: port 1(veth0f28c38) entered disabled state
[103874.537464] br-90d91c6c7798: port 1(veth38d25b7) entered blocking state
[103874.537468] br-90d91c6c7798: port 1(veth38d25b7) entered disabled state
[103874.537563] device veth38d25b7 entered promiscuous mode
[103874.537675] br-90d91c6c7798: port 1(veth38d25b7) entered blocking state
[103874.537678] br-90d91c6c7798: port 1(veth38d25b7) entered forwarding state
[103874.537964] br-90d91c6c7798: port 1(veth38d25b7) entered disabled state
[103874.764864] eth0: renamed from vethf781323
[103874.796840] IPv6: ADDRCONF(NETDEV_CHANGE): veth38d25b7: link becomes ready
[103874.796921] br-90d91c6c7798: port 1(veth38d25b7) entered blocking state
[103874.796926] br-90d91c6c7798: port 1(veth38d25b7) entered forwarding state

added amdkfd label

I can't find a warning at the exact line number. Probably because your kernel has some patches that aren't in v5.16. I suspect it's hitting this warning:

    if (WARN_ON(!work->func))
            return false;

Not sure how that's happening. kfd_process_notifier_release cancels a few workers. They're all initialized unconditionally during process creation. But I think there may be a problem if your kernel is compiled without HSA_AMD_SVM.

I think this later patch fixes the issue (the patch hasn't made it into the master branch yet, it's currently on drm-next):

ommit 0684944966cbec3edefaf773128500b5d4f4aea6 Author: Philip Yang Philip.Yang@amd.com Date: Wed Jan 12 11:50:50 2022 -0500

drm/amdkfd: svm range restore work deadlock when process exit

kfd_process_notifier_release flush svm_range_restore_work
which calls svm_range_list_lock_and_flush_work to flush deferred_list
work, but if deferred_list work mmput release the last user, it will
call exit_mmap -> notifier_release, it is deadlock with below backtrace.

Move flush svm_range_restore_work to kfd_process_wq_release to avoid
deadlock. Then svm_range_restore_work take task->mm ref to avoid mm is
gone while validating and mapping ranges to GPU.

Workqueue: events svm_range_deferred_list_work [amdgpu]
Call Trace:
 wait_for_completion+0x94/0x100
 __flush_work+0x12a/0x1e0
 __cancel_work_timer+0x10e/0x190
 cancel_delayed_work_sync+0x13/0x20
 kfd_process_notifier_release+0x98/0x2a0 [amdgpu]
 __mmu_notifier_release+0x74/0x1f0
 exit_mmap+0x170/0x200
 mmput+0x5d/0x130
 svm_range_deferred_list_work+0x104/0x230 [amdgpu]
 process_one_work+0x220/0x3c0

Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reported-by: Ruili Ji <ruili.ji@amd.com>
Tested-by: Ruili Ji <ruili.ji@amd.com>
Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>

Thanks for looking into this. I'm using a distro kernel, so it's quite possible there are patches applied. Concerning the config, this is what I get

$ grep -E 'HSA|SVM' /boot/config-$(uname -r)
CONFIG_HSA_AMD=y
CONFIG_INTEL_IOMMU_SVM=y

FWIW, I'm not sure how the AMD kernel drivers are laid out, but I don't even see an amdkfd module in /ib/modules/ for this kernel, I don't know if it's normal or not. I also tried installing the amdgpu module packaged by AMD, but apparently it doesn't build OOTB on this kernel.

KFD isn't its own module any more. It has been part of the amdgpu.ko module for a few years now. And yes, CONFIG_HSA_AMD_SVM is missing, which explains the problem. Other than the annoying warning, it should be harmless. The work struct is part of a 0-initialized structure, so there is no chance of random undefined behaviour.

Oh, OK, I thought the message might be related to the ROCm 4.5 OpenCL platform not recognizing my device and to ROCm 5 failing altogether on my hardware, but it looks like I'll have to look into those separately from this.

(Still, I should probably ask Debian to enable CONFIG_HSA_AMD_SVM anyway.)

mentioned in issue #2276 (closed)

Closing off due to the missing kernel CONFIG solving the issue

closed

WARNING in kernel/workqueue.c:3094 __flush_work.isra.0+0x20e/0x220 when accessing /dev/kfd

Brief summary of the problem:

Hardware description:

System information:

How to reproduce the issue:

Attached files:

Log files (for system lockups / game freezes / crashes)

Designs

Child items ...

Activity

Admin message

Admin message

WARNING in kernel/workqueue.c:3094 __flush_work.isra.0+0x20e/0x220 when accessing /dev/kfd

Brief summary of the problem:

Hardware description:

System information:

How to reproduce the issue:

Attached files:

Log files (for system lockups / game freezes / crashes)

Activity