igt@sysfs_heartbeat_interval@mixed@* - fail - CRITICAL: Failed assertion: sync_fence_status(spin->out_fence) == expect

https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8299/shard-skl10/igt@sysfs_heartbeat_interval@mixed@vecs0.html

Starting subtest: mixed
Starting dynamic subtest: vecs0
(sysfs_heartbeat_interval:1150) CRITICAL: Test assertion failure function client, file ../tests/i915/sysfs_heartbeat_interval.c:323:
(sysfs_heartbeat_interval:1150) CRITICAL: Failed assertion: sync_fence_status(spin->out_fence) == expect
(sysfs_heartbeat_interval:1150) CRITICAL: Good client: elapsed: 10.217ms, expected 1, got -5
Dynamic subtest vecs0 failed.
**** DEBUG ****
(sysfs_heartbeat_interval:1117) i915/gem_engine_topology-DEBUG: Test requirement passed: fstatat(engine, file, &st, 0) == 0
(sysfs_heartbeat_interval:1117) DEBUG: set preempt_timeout_ms:25
(sysfs_heartbeat_interval:1117) DEBUG: Test requirement passed: igt_sysfs_printf(engine, attr, "%u", value) > 0
(sysfs_heartbeat_interval:1117) DEBUG: Initial heartbeat_interval_ms:2500
(sysfs_heartbeat_interval:1117) igt_debugfs-DEBUG: Opening debugfs directory '/sys/kernel/debug/dri/0'
(sysfs_heartbeat_interval:1117) DEBUG: set heartbeat_interval_ms:25
(sysfs_heartbeat_interval:1117) DEBUG: Test requirement passed: igt_sysfs_printf(engine, attr, "%u", value) > 0
****  END  ****
Dynamic subtest vecs0: FAIL (0.799s)

This bug is split from #1459 (closed). Closing and archiving #1459 (closed).

added platform: EHL platform: GEN9 platform: KBL platform: SKL labels

mentioned in issue #1459 (closed)

The CI Bug Log issue associated to this bug has been updated by Lakshmi Vudum.

New filters associated

• GEN9 EHL: igt@sysfs_heartbeat_interval@mixed@* - fail - CRITICAL: Failed assertion: sync_fence_status(spin->out_fence) == expect (No new failures associated)

So what this test is checking is that with a mix of hanging and good clients, we only ever reset the hanging clients allowing the good ones to complete successfully.

Good client: elapsed: 10.217ms, expected 1, got -5

says that we reset an innocent batch. The heartbeat is 25ms and the batch length 10ms... So for whatever reason the preemption timeout is firing in the middle of the good client.

The CI Bug Log issue associated to this bug has been updated by Lakshmi Vudum.

New filters associated

• KBL: igt@sysfs_heartbeat_interval@mixed@vecs0 - fail - Failed assertion: sync_fence_status(spin->out_fence) == 0
  • https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_494/fi-kbl-soraka/igt@sysfs_heartbeat_interval@mixed@vecs0.html

mentioned in issue #1879 (closed)

A CI Bug Log filter associated to this bug has been updated by Lakshmi Vudum:

Description: GEN9 EHL: igt@sysfs_heartbeat_interval@mixed@* - fail/incomplete - CRITICAL: Failed assertion: sync_fence_status(spin->out_fence) == expect

Equivalent query: runconfig_tag IS IN ["DRM-TIP"] AND (machine_name IS IN ["shard-kbl3", "fi-glk-1", "shard-glkb1", "shard-glkb3", "shard-apl6", "fi-skl-6770hq", "fi-skl-6260u", "shard-apl2", "shard-kbl5", "fi-bxt-j4205", "shard-kbl4", "shard-glkb6", "shard-glkb2", "shard-kbl7", "fi-skl-6700hq", "fi-skl-6600u", "shard-apl5", "shard-apl7", "fi-bxt-dsi", "shard-glkb4", "shard-apl8", "shard-apl1", "fi-glk-dsi", "fi-kbl-7500u", "shard-kbl6", "shard-apl4", "shard-glkb5", "shard-kbl1", "fi-kbl-7560u", "shard-apl3", "fi-skl-6700k2", "fi-kbl-r", "fi-kbl-7567u", "fi-cfl-s2", "fi-skl-gvtdvm", "shard-kbl2", "fi-skl-guc", "fi-cfl-8700k", "fi-cfl-u", "fi-cfl-s3", "fi-glk-j4005", "shard-apl", "shard-glk", "shard-kbl", "shard-glk1", "shard-glk3", "shard-glk4", "shard-glk5", "shard-glk2", "shard-glkb", "shard-glk8", "shard-glk6", "shard-glk7", "pig-skl-6600", "pig-glk-j4005", "pig-glk-j5005", "fi-cfl-guc", "fi-kbl-guc", "fi-cfl-u2", "pig-skl-6260u", "fi-whl-u", "fi-kbl-x1275", "shard-glk9", "fi-cfl-8109u", "fi-skl-iommu", "fi-kbl-8809g", "fi-skl-caroline", "fi-apl-nasher", "fi-kbl-soraka", "shard-skl", "shard-skl1", "shard-skl2", "shard-skl3", "shard-skl4", "shard-skl5", "shard-skl6", "shard-skl7", "shard-skl8", "shard-skl9", "shard-skl10", "fi-apl-guc", "fi-skl-lmem", "fi-cml-u", "re-cml-u", "fi-cml-u2", "fi-cml-h", "fi-cml-s", "shard-apl0", "shard-glk0", "shard-kbl0", "shard-skl0", "fi-ehl-1", "re-ehl-1", "re-ehl-cham"] OR machine_tag IS IN ["EHL", "GEN9"]) AND ((testsuite_name = "IGT" AND test_name IS IN ["igt@sysfs_heartbeat_interval@mixed@rcs0", "igt@sysfs_heartbeat_interval@mixed@bcs0", "igt@sysfs_heartbeat_interval@mixed@vcs0", "igt@sysfs_heartbeat_interval@mixed@vecs0", "igt@sysfs_heartbeat_interval@mixed@vcs1"])) AND ((testsuite_name = "IGT" AND status_name IS IN ["incomplete", "fail"])) AND stderr ~= 'Failed assertion: sync_fence_status\(spin->out_fence\) == expect'

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_618/fi-kbl-7500u/igt@sysfs_heartbeat_interval@mixed@vcs0.html

Issue is not reproducing on re-ehl-1 in CI resume runs from DRM_8163 to DRM_9059 .

• EHL logs : http://gfx-ci.fi.intel.com/cibuglog-ng/results/all?query_key=0756c6ca8637c11b5d11cece679c7d773e25b141

Issue is not reproducing on KBL,SKL,APL,GLK

• shard logs : http://gfx-ci.fi.intel.com/tree/drm-tip/shards-all.html?testfilter=igt@sysfs_heartbeat_interval@mixed@

Issue is not reproducing on CML,CFL,BXT and WHL in CI resume run. Passing logs attached below

• http://gfx-ci.fi.intel.com/cibuglog-ng/results/all?query_key=ca0f7fcfaffbe0480f2fecd6600b26c8dc34d8b4
• http://gfx-ci.fi.intel.com/cibuglog-ng/results/all?query_key=9aa4bd41f209fc50dc2f8a34273c82e870b53016
• http://gfx-ci.fi.intel.com/cibuglog-ng/results/all?query_key=22c81c1e2e82561cfe388670cd9c38447e86e3b2
• http://gfx-ci.fi.intel.com/cibuglog-ng/results/all?query_key=36214f558537aa9f39003500f6e09a666d952ca7

Hence closing this bug id #1731 (closed)

closed

This issue is still happening very regularly, so re-opening the issue.

reopened

Issue is reproducing on KBL

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_723/fi-kbl-soraka/igt@sysfs_heartbeat_interval@mixed@vcs0.html

Issue is reproducing on SKL

• https://intel-gfx-ci.01.org/tree/drm-tip/IGT_5891/shard-skl6/igt@sysfs_heartbeat_interval@mixed@rcs0.html

A CI Bug Log filter associated to this bug has been updated by :

Description: BDW GEN9 EHL: igt@sysfs_heartbeat_interval@mixed@* - fail/incomplete - CRITICAL: Failed assertion: sync_fence_status(spin->out_fence) == expect

Equivalent query: runconfig_tag IS IN ["DRM-TIP"] AND (machine_name IS IN ["shard-kbl3", "fi-glk-1", "shard-glkb1", "shard-glkb3", "shard-apl6", "fi-skl-6770hq", "fi-bdw-gvtdvm", "fi-skl-6260u", "shard-apl2", "shard-kbl5", "fi-bxt-j4205", "shard-kbl4", "shard-glkb6", "shard-glkb2", "shard-kbl7", "fi-skl-6700hq", "fi-skl-6600u", "shard-apl5", "shard-apl7", "fi-bxt-dsi", "fi-bdw-5557u", "shard-glkb4", "shard-apl8", "shard-apl1", "fi-glk-dsi", "fi-kbl-7500u", "shard-kbl6", "shard-apl4", "shard-glkb5", "shard-kbl1", "fi-kbl-7560u", "shard-apl3", "fi-skl-6700k2", "fi-kbl-r", "fi-kbl-7567u", "fi-cfl-s2", "fi-skl-gvtdvm", "shard-kbl2", "fi-skl-guc", "fi-cfl-8700k", "fi-cfl-u", "fi-cfl-s3", "fi-glk-j4005", "shard-apl", "shard-glk", "shard-kbl", "shard-glk1", "shard-glk3", "shard-glk4", "shard-glk5", "shard-glk2", "shard-glkb", "shard-glk8", "shard-glk6", "shard-glk7", "pig-skl-6600", "pig-glk-j4005", "pig-glk-j5005", "fi-cfl-guc", "fi-kbl-guc", "fi-cfl-u2", "pig-skl-6260u", "fi-whl-u", "fi-kbl-x1275", "shard-glk9", "fi-cfl-8109u", "fi-skl-iommu", "fi-kbl-8809g", "fi-skl-caroline", "fi-apl-nasher", "fi-bdw-samus", "fi-kbl-soraka", "shard-skl", "shard-skl1", "shard-skl2", "shard-skl3", "shard-skl4", "shard-skl5", "shard-skl6", "shard-skl7", "shard-skl8", "shard-skl9", "shard-skl10", "fi-apl-guc", "fi-skl-lmem", "fi-cml-u", "re-cml-u", "fi-cml-u2", "fi-cml-h", "fi-cml-s", "shard-apl0", "shard-glk0", "shard-kbl0", "shard-skl0", "fi-ehl-1", "re-ehl-1", "re-ehl-cham", "hbd-ehl-1"] OR machine_tag IS IN ["EHL", "BDW", "GEN9"]) AND ((testsuite_name = "IGT" AND test_name IS IN ["igt@sysfs_heartbeat_interval@mixed@rcs0", "igt@sysfs_heartbeat_interval@mixed@bcs0", "igt@sysfs_heartbeat_interval@mixed@vcs0", "igt@sysfs_heartbeat_interval@mixed@vecs0", "igt@sysfs_heartbeat_interval@mixed@vcs1"])) AND ((testsuite_name = "IGT" AND status_name IS IN ["incomplete", "fail"])) AND stderr ~= 'Failed assertion: sync_fence_status\(spin->out_fence\) == expect'

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_761/fi-bdw-gvtdvm/igt@sysfs_heartbeat_interval@mixed@bcs0.html

added platform: BDW label

added platform: BSW/CHT label

A CI Bug Log filter associated to this bug has been updated by :

Description: BSW BDW GEN9 EHL: igt@sysfs_heartbeat_interval@mixed@* - fail/incomplete - CRITICAL: Failed assertion: sync_fence_status(spin->out_fence) == expect

Equivalent query: runconfig_tag IS IN ["DRM-TIP"] AND (machine_name IS IN ["shard-kbl3", "fi-glk-1", "shard-glkb1", "shard-glkb3", "shard-apl6", "fi-skl-6770hq", "fi-bdw-gvtdvm", "fi-skl-6260u", "shard-apl2", "shard-kbl5", "fi-bxt-j4205", "shard-kbl4", "shard-glkb6", "shard-glkb2", "shard-kbl7", "fi-skl-6700hq", "fi-skl-6600u", "shard-apl5", "shard-apl7", "fi-bxt-dsi", "fi-bdw-5557u", "shard-glkb4", "shard-apl8", "fi-bsw-n3050", "shard-apl1", "fi-glk-dsi", "fi-kbl-7500u", "shard-kbl6", "shard-apl4", "shard-glkb5", "shard-kbl1", "fi-kbl-7560u", "shard-apl3", "fi-skl-6700k2", "fi-kbl-r", "fi-kbl-7567u", "fi-cfl-s2", "fi-skl-gvtdvm", "shard-kbl2", "fi-skl-guc", "fi-cfl-8700k", "fi-cfl-u", "fi-cfl-s3", "fi-glk-j4005", "shard-apl", "shard-glk", "shard-kbl", "shard-glk1", "shard-glk3", "shard-glk4", "shard-glk5", "shard-glk2", "shard-glkb", "shard-glk8", "shard-glk6", "shard-glk7", "pig-skl-6600", "pig-glk-j4005", "pig-glk-j5005", "fi-bsw-cyan", "fi-cfl-guc", "fi-kbl-guc", "fi-cfl-u2", "pig-skl-6260u", "fi-whl-u", "fi-kbl-x1275", "shard-glk9", "fi-cfl-8109u", "fi-skl-iommu", "fi-kbl-8809g", "fi-bsw-kefka", "fi-skl-caroline", "fi-apl-nasher", "fi-bdw-samus", "fi-kbl-soraka", "shard-skl", "shard-skl1", "shard-skl2", "shard-skl3", "shard-skl4", "shard-skl5", "shard-skl6", "shard-skl7", "shard-skl8", "shard-skl9", "shard-skl10", "fi-apl-guc", "fi-skl-lmem", "fi-cml-u", "re-cml-u", "fi-cml-u2", "fi-cml-h", "fi-cml-s", "shard-apl0", "shard-glk0", "shard-kbl0", "shard-skl0", "fi-bsw-nick", "fi-ehl-1", "re-ehl-1", "re-ehl-cham", "hbd-ehl-1"] OR machine_tag IS IN ["EHL", "BDW", "BSW", "GEN9"]) AND ((testsuite_name = "IGT" AND test_name IS IN ["igt@sysfs_heartbeat_interval@mixed@rcs0", "igt@sysfs_heartbeat_interval@mixed@bcs0", "igt@sysfs_heartbeat_interval@mixed@vcs0", "igt@sysfs_heartbeat_interval@mixed@vecs0", "igt@sysfs_heartbeat_interval@mixed@vcs1"])) AND ((testsuite_name = "IGT" AND status_name IS IN ["incomplete", "fail"])) AND stderr ~= 'Failed assertion: sync_fence_status\(spin->out_fence\) == expect'

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_766/fi-bsw-kefka/igt@sysfs_heartbeat_interval@mixed@vecs0.html
• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_766/fi-bsw-n3050/igt@sysfs_heartbeat_interval@mixed@vcs0.html

Issue is reproducing on SKL. Log: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10692/shard-skl2/igt@sysfs_heartbeat_interval@mixed@rcs0.html

Issue is not reproducing on KBL, Last seen on KBL 1 month,3 weeks ago. Hence removing KBL tag. Log: https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_920/fi-kbl-soraka/igt@sysfs_heartbeat_interval@mixed@bcs0.html

Issue is not reproducing on EHL. hence removing EHL tag. Log: https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_915/fi-ehl-2/igt@sysfs_heartbeat_interval@mixed@bcs0.html

Issue is reproducing on BSW. Last seen on BSW ~4 weeks ago. Hence removing BSW tag Log:https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_920/fi-bsw-kefka/igt@sysfs_heartbeat_interval@mixed@vcs0.html

Issue is reproducing sporadically on BDW twice in a month. Log: https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_908/fi-bdw-gvtdvm/igt@sysfs_heartbeat_interval@mixed@bcs0.html

removed platform: BSW/CHT platform: EHL platform: KBL labels

A CI Bug Log filter associated to this bug has been updated by :

Description: BSW BDW GEN9 EHL: igt@sysfs_heartbeat_interval@mixed@* - fail/incomplete - CRITICAL: Failed assertion: sync_fence_status(spin->out_fence) == expect

Equivalent query: runconfig_tag IS IN ["DRM-TIP"] AND (machine_name IS IN ["shard-skl", "shard-skl1", "shard-skl2", "shard-skl3", "shard-skl4", "shard-skl5", "shard-skl6", "fi-skl-6770hq", "fi-bdw-gvtdvm", "shard-skl7", "shard-skl8", "fi-skl-6260u", "shard-skl10", "shard-skl9", "fi-skl-lmem", "fi-skl-6700hq", "fi-skl-6600u", "fi-bdw-5557u", "fi-skl-6700k2", "fi-skl-gvtdvm", "fi-skl-guc", "fi-skl-iommu", "fi-skl-caroline", "fi-bdw-samus"] OR machine_tag IS IN ["EHL", "BDW", "BSW", "GEN9KL"]) AND ((testsuite_name = "IGT" AND test_name IS IN ["igt@sysfs_heartbeat_interval@mixed@rcs0", "igt@sysfs_heartbeat_interval@mixed@bcs0", "igt@sysfs_heartbeat_interval@mixed@vcs0", "igt@sysfs_heartbeat_interval@mixed@vecs0", "igt@sysfs_heartbeat_interval@mixed@vcs1"])) AND ((testsuite_name = "IGT" AND status_name IS IN ["incomplete", "fail"])) AND stderr ~= 'Failed assertion: sync_fence_status\(spin->out_fence\) == expect'

No new failures caught with the new filter