[CI][BAT] [CML only] igt@* - incomplete - timeout/system hang?

Submitted by Martin Peres @mupuf

Assigned to Intel GFX Bugs mailing list

Link to original bug (#110566)

Description

added CI feature: CI Infra platform: CML priority::highest severity::normal + 1 deleted label

CI Bug Log said:

The CI Bug Log issue associated to this bug has been updated.

New filters associated

* CML: random tests - incomplete
- https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_267/fi-cml-u/igt@kms_busy@basic-modeset-c.html
- https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_269/fi-cml-u/igt@kms_plane@pixel-format-pipe-b-planes.html
- https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6001/re-cml-u/igt@kms_flip@flip-vs-panning-interruptible.html
- https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_270/fi-cml-u/igt@kms_cursor_crc@cursor-128x42-random.html
- https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_270/fi-cml-u/igt@gem_exec_big@single.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

CML: random tests - incomplete
CML: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6010/re-cml-u/igt@gem_tiled_wb.html
  
• https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6012/re-cml-u/igt@gem_tiled_wb.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

CML: random tests - incomplete
CML: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6062/re-cml-u/igt@kms_chamelium@common-hpd-after-suspend.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

CML: random tests - incomplete
CML: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6061/re-cml-u/igt@kms_cursor_crc@cursor-64x64-offscreen.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

CML: random tests - incomplete
CML: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6067/re-cml-u/igt@kms_flip@flip-vs-fences.html
  
• https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6067/re-cml-u/igt@kms_psr@psr2_sprite_plane_move.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

CML: random tests - incomplete
CML: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6072/re-cml-u/igt@gem_pwrite@big-gtt-backwards.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

CML: random tests - incomplete
CML: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_285/fi-cml-u/igt@gem_pwrite@big-cpu-backwards.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

CML: random tests - incomplete
CML: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6085/re-cml-u/igt@i915_pm_rpm@pm-caching.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

CML: random tests - incomplete
CML: all tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_288/fi-cml-u/igt@kms_frontbuffer_tracking@psr-2p-scndscrn-pri-shrfb-draw-blt.html
  
• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_288/fi-cml-u/igt@kms_cursor_crc@pipe-b-cursor-128x128-onscreen.html
  
• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_289/fi-cml-u/igt@kms_vblank@pipe-c-wait-busy.html
  
• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_289/fi-cml-u/igt@kms_vblank@pipe-c-ts-continuation-modeset-rpm.html
  
• https://intel-gfx-ci.01.org/tree/drm-tip/IGT_5003/fi-cml-u/igt@core_auth@basic-auth.html
  
• https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6121/re-cml-u/igt@gem_exec_parallel@bsd.html
  
• https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6121/re-cml-u/igt@i915_pm_rpm@gem-execbuf.html
  
• https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6121/re-cml-u/igt@kms_plane@pixel-format-pipe-b-planes-source-clamping.html
  
• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_292/fi-cml-u/igt@kms_plane_scaling@pipe-b-scaler-with-clipping-clamping.html
  
• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_292/fi-cml-u/igt@gem_exec_reloc@basic-wc-gtt-noreloc.html
  
• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_292/fi-cml-u2/igt@kms_cursor_crc@pipe-b-cursor-256x85-random.html

Martin Peres @mupuf said:

Putting to highest priority, since incompletes are now everywhere!

Stanislav Lisovskiy said:

Different tests, but always same 2 machines(fi-cml-u, re-cml-u) - kinda suspicious. May be it would make sense to take it from CI to run the tests manually and check?

Jani Saarinen @jani.saarinen said:

Lakhsmi, is this still valid?

LAKSHMINARAYANA VUDUM @l4kshmi said:

(In reply to Jani Saarinen from comment 12)

Lakhsmi, is this still valid?
Yes, it's still happening. But I see that Jenkins gives up in some cases.

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_378/fi-cml-s/igt@gem_ctx_switch@vcs0-heavy.html

Starting subtest: vcs0-heavy
[32.199975] Abort requested by sudo LD_LIBRARY_PATH=/opt/igt/lib:/opt/igt/lib/x86_64-linux-gnu IGT_CI_META_TEST=yes IGT_REBOOT_ON_FATAL_ERROR=yes stdb [1183] via Terminated, terminating children
[32.200923] Abort requested by /sbin/init 3 [1] via Hangup, terminating children
[32.201059] Abort requested by sudo LD_LIBRARY_PATH=/opt/igt/lib:/opt/igt/lib/x86_64-linux-gnu IGT_CI_META_TEST=yes IGT_REBOOT_ON_FATAL_ERROR=yes stdb [1183] via Hangup, terminating children
[32.212075] Closing watchdogs
Done.

We need to go through all the failures and see if we need to split the issue.

Vanshidhar Konda said:

I looked over about 20 logs for this issue. For about 60% of the logs there is a line saying: This is power.sh, remotely rebooting this machine.

It seems like the machine was reset remotely during the test - in a number of cases while the test was executing.

Jani Saarinen @jani.saarinen said:

Tomi, is there something can be done to CI for this ?

LAKSHMINARAYANA VUDUM @l4kshmi said:

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_396/fi-cml-s/igt_runner6.txt
Subtest pipe-A-overlay-size-64: SUCCESS (2.454s)
[37.463866] [11/99] (945s left) kms_plane (plane-panning-top-left-pipe-b-planes)
Starting subtest: plane-panning-top-left-pipe-B-planes
[41.235917] Abort requested by /sbin/init 3 [1] via Hangup, terminating children
[41.237585] Abort requested by sudo LD_LIBRARY_PATH=/opt/igt/lib:/opt/igt/lib/x86_64-linux-gnu IGT_CI_META_TEST=yes IGT_REBOOT_ON_FATAL_ERROR=yes stdb [1215] via Terminated, terminating children
[42.417619] Closing watchdogs
Done.
These incompletes should be fixed as part of bug 111747.

Seems to be gfx-ci/i915-infra#80 (closed)

moved to gfx-ci/i915-infra#83

The CI Bug Log issue associated to this bug has been archived.

New failures matching the above filters will not be associated to this bug anymore.