igt@gem_exec_balancer@hang - fail - Failed assertion: sync_fence_status(c->spin[1]->out_fence) == -5, error: 0 != -5

https://intel-gfx-ci.01.org/tree/drm-tip/IGT_5449/shard-tglb5/igt@gem_exec_balancer@hang.html

Starting subtest: hang
(gem_exec_balancer:1379) CRITICAL: Test assertion failure function hangme, file ../tests/i915/gem_exec_balancer.c:1705:
(gem_exec_balancer:1379) CRITICAL: Failed assertion: sync_fence_status(c->spin[1]->out_fence) == -5
(gem_exec_balancer:1379) CRITICAL: error: 0 != -5
Subtest hang failed.

added CI feature: GEM platform: ICL platform: TGL labels

changed the description

The CI Bug Log issue associated to this bug has been updated by Lakshmi Vudum.

New filters associated

• ICL TGL: igt@gem_exec_balancer@hang - fail - Failed assertion: sync_fence_status(c->spin[1]->out_fence) == -5, error: 0 != -5
  • https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_16603/shard-tglb5/igt@gem_exec_balancer@hang.html
  • https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_16613/shard-tglb1/igt@gem_exec_balancer@hang.html
  • https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_16614/shard-tglb6/igt@gem_exec_balancer@hang.html
  • https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_16616/shard-tglb5/igt@gem_exec_balancer@hang.html
  • https://intel-gfx-ci.01.org/tree/drm-tip/IGT_5449/shard-tglb5/igt@gem_exec_balancer@hang.html
  • https://intel-gfx-ci.01.org/tree/drm-tip/IGT_5450/shard-iclb1/igt@gem_exec_balancer@hang.html
  • https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4185/shard-iclb4/igt@gem_exec_balancer@hang.html

closed via commit igt-gpu-tools@a378158d

commit a378158d35b994e2ded0287dfe2770b8345b573c (HEAD, upstream/master)
Author: Chris Wilson <chris@chris-wilson.co.uk>
Date:   Sat Feb 29 11:17:16 2020 +0000

    i915/gem_exec_balancer: Bump priority of background load for hang detection
    
    If we ensure that the background load has elevated priority, it will
    kick off the preempt-timeout hang detection, ensuring that our hang
    detection is fast.
    
    Closes: https://gitlab.freedesktop.org/drm/intel/issues/1277
    Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
    Acked-by: Mika Kuoppala <mika.kuoppala@linux.intel.com>

This only works for -ETIME which is one of the failures I was looking at.

reopened

The issue is still seen in re-tgl-u2 with CI_DRM_8044 and IGT_5483. The patch was merged in IGT_5479 https://intel-gfx-ci.01.org/tree/drm-tip/IGT_5479/git-log-oneline.txt

Not the same issue.

No, I was thinking this was the gem_bo_busy() assert from 8041.

closed

reopened

commit 8e9f84cf5cac248a1c6a5daa4942879c8b765058 (HEAD -> drm-intel-next-queued, drm-intel/for-linux-next, drm-intel/drm-intel-next-queued)
Author: Chris Wilson <chris@chris-wilson.co.uk>
Date:   Wed Mar 4 12:18:49 2020 +0000

    drm/i915/gt: Propagate change in error status to children on unhold
    
    As we release the head requests back into the queue, propagate any
    change in error status that may have occurred while the requests were
    temporarily suspended.
    
    Closes: https://gitlab.freedesktop.org/drm/intel/issues/1277
    Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
    Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
    Reviewed-by: Mika Kuoppala <mika.kuoppala@linux.intel.com>
    Link: https://patchwork.freedesktop.org/patch/msgid/20200304121849.2448028-2-chris@chris-wilson.co.uk

That should paper over any gap, I hope.

closed

I think this is till happening, see https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8182/shard-tglb1/igt@gem_exec_balancer@hang.html

reopened

added platform: BDW label

A CI Bug Log filter associated to this bug has been updated by Lakshmi Vudum:

Description: BDW ICL TGL: igt@gem_exec_balancer@hang - fail - Failed assertion: sync_fence_status(c->spin[1]->out_fence) == -5, error: 0 != -5

Equivalent query: runconfig_tag IS IN ["DRM-TIP"] AND (machine_name IS IN ["fi-bdw-gvtdvm", "fi-icl-u2", "shard-iclb", "shard-iclb1", "shard-iclb2", "shard-iclb3", "shard-iclb4", "shard-iclb5", "shard-iclb6", "fi-icl-u3", "shard-iclb7", "fi-icl-y", "shard-iclb8", "fi-icl-guc", "fi-icl-dsi", "re-icl-u", "fi-icl-u4", "fi-tgl-u", "fi-bdw-5557u", "re-tgl-u", "fi-tgl-u2", "shard-tglb1", "shard-tglb2", "shard-tglb3", "shard-tglb4", "shard-tglb5", "shard-tglb6", "re-tgl1-display", "re-tgl2-display", "shard-tglb7", "shard-tglb8", "shard-tglb0", "fi-tgl-y", "re-tgl-y", "shard-tglb9", "re-tgl-cham", "fi-icl-1065g7", "fi-tgl-guc", "re-tgl-u2", "fi-tgl-dsi", "ba-tgl-1", "fi-icl-u", "fi-bdw-samus"] OR machine_tag IS IN ["ICL", "BDW", "TGL"]) AND ((testsuite_name = "IGT" AND test_name IS IN ["igt@gem_exec_balancer@hang"])) AND ((testsuite_name = "IGT" AND status_name IS IN ["fail"])) AND stderr ~= 'Failed assertion: sync_fence_status\(c->spin\[1\]->out_fence\) == -5\n.*error: 0 != -5'

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_458/fi-bdw-5557u/igt@gem_exec_balancer@hang.html

closed via commit igt-gpu-tools@9a94fb4b

Ok, seems to be working theory:

We are detecting the hang before we queue the continuation, changing the semantics of the test.

commit 9a94fb4b5ebb82185fb52a1709366458597a62ab (HEAD, upstream/master)
Author: Chris Wilson <chris@chris-wilson.co.uk>
Date:   Thu Apr 30 11:58:11 2020 +0100

    i915/gem_exec_balancer: Queue all hangs together
    
    Just in case we detect the invalid CS before we are all setup in the
    test to check for the expected results, use a fence to submit all the
    hanging spinners together.
    
    Closes: https://gitlab.freedesktop.org/drm/intel/-/issues/1277
    Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
    Acked-by: Mika Kuoppala <mika.kuoppala@linux.intel.com>

The CI Bug Log issue associated to this bug has been archived.

New failures matching the above filters will not be associated to this bug anymore.