igt@xe_wedged@basic-wedged - dmesg-warn - *ERROR* GT0: reset failed (-ECANCELED)
<6> [438.651583] xe 0000:00:02.0: [drm] GT0: reset queued
<6> [438.652148] xe 0000:00:02.0: [drm] GT0: reset started
<5> [438.652170] FAULT_INJECTION: forcing a failure.
name fail_gt_reset, interval 1, probability 100, space 0, times 1
<4> [438.652174] CPU: 2 UID: 0 PID: 2228 Comm: kworker/u32:12 Not tainted 6.12.0-rc1-xe #1
<4> [438.652176] Hardware name: Intel Corporation Lunar Lake Client Platform/LNL-M LP5 RVP1, BIOS LNLMFWI1.R00.3221.D89.2409021714 09/02/2024
<4> [438.652177] Workqueue: gt-ordered-wq gt_reset_worker [xe]
<4> [438.652236] Call Trace:
<4> [438.652236] <TASK>
<4> [438.652238] dump_stack_lvl+0xa3/0xc0
<4> [438.652244] dump_stack+0x10/0x20
<4> [438.652246] should_fail_ex+0x16f/0x180
<4> [438.652250] should_fail+0xb/0x20
<4> [438.652251] gt_reset_worker+0x76/0x1d0 [xe]
<4> [438.652289] process_one_work+0x21c/0x730
<4> [438.652294] worker_thread+0x1db/0x3a0
<4> [438.652296] ? __pfx_worker_thread+0x10/0x10
<4> [438.652298] kthread+0xf3/0x130
<4> [438.652301] ? __pfx_kthread+0x10/0x10
<4> [438.652303] ret_from_fork+0x3a/0x60
<4> [438.652305] ? __pfx_kthread+0x10/0x10
<4> [438.652307] ret_from_fork_asm+0x1a/0x30
<4> [438.652312] </TASK>
<3> [438.652313] xe 0000:00:02.0: [drm] *ERROR* GT0: reset failed (-ECANCELED)
<3> [438.652528] xe 0000:00:02.0: [drm] *ERROR* CRITICAL: Xe has declared device 0000:00:02.0 as wedged.
IOCTLs and executions are blocked. Only a rebind may clear the failure
Please file a _new_ bug report at https://gitlab.freedesktop.org/drm/xe/kernel/issues/new
<7> [439.661870] xe 0000:00:02.0: [drm:intel_power_well_enable [xe]] enabling DC_off
<7> [439.662071] xe 0000:00:02.0: [drm:gen9_set_dc_state.part.0 [xe]] Setting DC state from 02 to 00
<7> [439.662220] xe 0000:00:02.0: [drm:intel_power_well_enable [xe]] enabling PW_2
<7> [439.662429] xe 0000:00:02.0: [drm:intel_power_well_enable [xe]] enabling PW_B
<7> [439.662999] xe 0000:00:02.0: [drm:intel_power_well_enable [xe]] enabling PW_C
<7> [439.663122] xe 0000:00:02.0: [drm:intel_power_well_enable [xe]] enabling PW_D
<7> [439.663234] xe 0000:00:02.0: [drm:intel_power_well_enable [xe]] enabling PICA_TC
<7> [439.721217] xe 0000:00:02.0: [drm:atomic_remove_fb [drm]] Disabling [PLANE:32:plane 1A] because [FB:226] is removed
<7> [439.721796] xe 0000:00:02.0: [drm:skl_compute_wm [xe]] [PLANE:32:plane 1A] ddb ( 0 - 4054) -> ( 0 - 0), size 4054 -> 0
<7> [439.722078] xe 0000:00:02.0: [drm:skl_compute_wm [xe]] [PLANE:32:plane 1A] level *wm0,*wm1,*wm2,*wm3,*wm4,*wm5, wm6, wm7,*twm,*swm,*stwm -> wm0, wm1, wm2, wm3, wm4, wm5, wm6, wm7, twm, swm, stwm
<7> [439.722319] xe 0000:00:02.0: [drm:skl_compute_wm [xe]] [PLANE:32:plane 1A] lines 1, 12, 12, 12, 11, 12, 0, 0, 0, 3, 0 -> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
<7> [439.722548] xe 0000:00:02.0: [drm:skl_compute_wm [xe]] [PLANE:32:plane 1A] blocks 24, 289, 289, 289, 265, 289, 0, 0, 38, 73, 87 -> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
<7> [439.722769] xe 0000:00:02.0: [drm:skl_compute_wm [xe]] [PLANE:32:plane 1A] min_ddb 28, 319, 319, 319, 293, 319, 0, 0, 39, 82, 88 -> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
<7> [439.723035] xe 0000:00:02.0: [drm:intel_bw_atomic_check [xe]] [CRTC:82:pipe A] data rate 0 num active planes 0
<7> [439.723322] xe 0000:00:02.0: [drm:intel_bw_atomic_check [xe]] QGV point 0: max bw 23530 required 0 qgv_peak_bw: 38400
<7> [439.723599] xe 0000:00:02.0: [drm:intel_bw_atomic_check [xe]] QGV point 1: max bw 36260 required 0 qgv_peak_bw: 38400Activity
added feature: GEM platform: ADL_P platform: BMG platform: DG2 platform: LNL labels
The test is running as expected and passing, but since the injected failure also triggers a dump_stack() in the kernel, igt_runner incorrectly believes this to be a real dmesg error and flags the test as "dmesg_warn."
We can avoid this by explicitly setting the verbosity setting for the gt reset injection interface to "1" (instead of leaving it at the default "2") --- the general fault injection and wedged messages will still be printed (which the test can properly tell igt_runner to ignore), but the unnecessary stack trace will be skipped.
I sent a fix for this at https://patchwork.freedesktop.org/series/141810/, but then noticed @kxkamil already has a different kind of fix at https://patchwork.freedesktop.org/patch/626241/?series=140511&rev=5 that instead lets the dump_stack() go through but tries to ignore more regular expressions that show up there.
mentioned in issue #3119 (closed)
This is a good patch, you have my r-b for it here until I manage to revive my machine so you can add: Reviewed-by: Kamil Konieczny kamil.konieczny@linux.intel.com
A CI Bug Log filter associated to this bug has been updated by Vinay.
Description: PVC ADL_P BMG DG2 LNL: igt@xe_wedged@basic-wedged - dmesg-warn - *ERROR* GT0: reset failed (-ECANCELED)
Equivalent query: runconfig_tag IS IN ["xe"] AND machine_tag IS IN ["DG2", "PVC", "LNL", "BMG", "ADL-P"] AND ((testsuite_name = "IGT" AND test_name IS IN ["igt@xe_wedged@basic-wedged"])) AND ((testsuite_name = "IGT" AND status_name IS IN ["dmesg-warn"])) AND dmesg ~= '\*ERROR\* GT0: reset failed \(-ECANCELED\)'
New failures caught by the filter:
added platform: PVC label
mentioned in commit ifdu/igt-gpu-tools@58275182
mentioned in commit ifdu/igt-gpu-tools@21076ee0
It seems like the reset failure errors are being correctly masked, so we aren't hitting this issue per se. Instead, it seems like the CI filter is picking up a group of other warnings and is associating them with this issue. Namely:
<3> [557.431647] xe 0000:00:02.0: [drm] ERROR AUX B/DDI B/PHY B: did not complete or timeout within 10ms (status 0xad40023f)
Whether this is a correct association or not is debatable, but the observed error above can also be masked if needed.
The recent failures being reported against this test are on ADL-P and BMG, and the reason for each seems to be different.
- As @jcavitt mentioned, the recent ADL issues are display timeouts, which are already tracked by many other tickets (#3086 (closed), #2953, #2525 (closed), #2283 (closed), #1621 (closed), etc)
- The recent BMG failures have a different, complaining about AUDIO_PLAYBACK use count. Those are already being tracked by #3468 (closed).
The original issue here does seem to be gone, so re-closing. The continuing failures noted above should hopefully get captured by the proper tickets once this one gets out of the way and stops grabbing the results first.
