[CI][DRMTIP] Time is passing at a different rate between IGT machines and the controller

Submitted by Martin Peres

Assigned to Default DRI bug account

Link to original bug (#109607)

Description

This leads to bug like this:

[78/80] (157s left) kms_flip (2x-flip-vs-modeset-vs-hang)
Build timed out (after 20 minutes). Marking the build as aborted.

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_206/fi-icl-u3/igt@kms_flip@2x-flip-vs-modeset-vs-hang.html

added priority::medium severity::normal + 1 deleted label

CI Bug Log said:

The CI Bug Log issue associated to this bug has been updated.

New filters associated

* fi-icl-u3: random tests - incomplete
- https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_212/fi-icl-u3/igt@kms_plane_scaling@pipe-b-scaler-with-clipping-clamping.html
- https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_214/fi-icl-u3/igt@kms_plane_scaling@pipe-b-scaler-with-clipping-clamping.html
- https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_215/fi-icl-u3/igt@kms_plane_scaling@pipe-b-scaler-with-clipping-clamping.html
- https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_205/fi-icl-u3/igt@kms_flip@2x-flip-vs-modeset-vs-hang.html
- https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_206/fi-icl-u3/igt@kms_flip@2x-flip-vs-modeset-vs-hang.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-icl-u3: random tests - incomplete
fi-icl-u3: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_217/fi-icl-u3/igt@kms_plane_scaling@pipe-c-scaler-with-clipping-clamping.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-icl-u3: random tests - incomplete
fi-icl-u2, fi-icl-u3: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_218/fi-icl-u2/igt@kms_plane@pixel-format-pipe-c-planes.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-icl-u2, fi-icl-u3: random tests - incomplete
fi-icl-u2, fi-icl-u3: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_221/fi-icl-u2/igt@kms_plane@pixel-format-pipe-a-planes.html
  
• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_221/fi-icl-u2/igt@kms_plane@pixel-format-pipe-b-planes.html
  
• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_221/fi-icl-u3/igt@kms_plane@pixel-format-pipe-a-planes.html
  
• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_221/fi-icl-u3/igt@kms_plane@pixel-format-pipe-b-planes.html
  
• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_222/fi-icl-u2/igt@kms_plane@pixel-format-pipe-b-planes.html
  
• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_223/fi-icl-u2/igt@kms_plane@pixel-format-pipe-b-planes.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-icl-u2, fi-icl-u3: random tests - incomplete
fi-icl-u2, fi-icl-u3: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_225/fi-icl-u3/igt@kms_rotation_crc@multiplane-rotation-cropping-top.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-icl-u2, fi-icl-u3: random tests - incomplete
fi-icl-u2, fi-icl-u3: random tests - incomplete

No new failures caught with the new filter

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-icl-u2, fi-icl-u3: random tests - incomplete
fi-icl-u2, fi-icl-u3: random tests - incomplete

No new failures caught with the new filter

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-icl-u2, fi-icl-u3: random tests - incomplete
fi-kbl-7560u fi-icl-u2, fi-icl-u3: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_228/fi-kbl-7560u/igt@kms_plane@pixel-format-pipe-c-planes-source-clamping.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-kbl-7560u fi-icl-u2, fi-icl-u3: random tests - incomplete
fi-kbl-7560u fi-icl-u2, fi-icl-u3: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_243/fi-icl-u3/igt@kms_flip@2x-blocking-wf_vblank.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-kbl-7560u fi-icl-u2, fi-icl-u3: random tests - incomplete
fi-kbl-7560u fi-icl-u2, fi-icl-u3: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_251/fi-icl-u2/igt@gem_mmap_gtt@big-copy-xy.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-kbl-7560u fi-icl-u2, fi-icl-u3: random tests - incomplete
fi-kbl-7560u, fi-cfl-8109u, fi-icl-u2, fi-icl-u3: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_252/fi-cfl-8109u/igt@kms_plane_lowres@pipe-b-tiling-none.html

Arek Hiler said:

This is an infrastructure deficiency coming from technical limitations. It is especially bad as it requires us to create a catch-all bug for incompletes, that may obscure some serious problems.

To mitigate that we look at all the failures under this bug periodically, and if we see anything that looks suspicious we file it separately.

This generates manual labor and increases latency in detecting new issues causing incomplete.

This is important to fix from CI infrastructure point of view.

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-kbl-7560u, fi-cfl-8109u, fi-icl-u2, fi-icl-u3: random tests - incomplete
fi-kbl-7560u, fi-cfl-8109u, fi-icl-u2, fi-icl-u3: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_302/fi-kbl-7560u/igt@kms_vblank@pipe-c-wait-forked-hang.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-kbl-7560u, fi-cfl-8109u, fi-icl-u2, fi-icl-u3: random tests - incomplete
fi-kbl-7560u, fi-cfl-8109u, fi-icl-u2, fi-icl-u3: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_302/fi-kbl-7560u/igt@kms_vblank@pipe-a-wait-idle.html

CI Bug Log said:

The CI Bug Log issue associated to this bug has been updated.

New filters associated

* shard-iclb5: igt@kms_flip@flip-vs-modeset-vs-hang-interruptible - dmesg-warn - watchdog: BUG: soft lockup - CPU#\d stuck for \d+s!
- https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6719/shard-iclb5/igt@kms_flip@flip-vs-modeset-vs-hang-interruptible.html

A CI Bug Log filter associated to this bug has been updated by :

Description: fi-cfl-guc,fi-kbl-7560u, fi-cfl-8109u, fi-icl-u2, fi-icl-u3: random tests - incomplete

Equivalent query: runconfig_tag IS IN ["DRM-TIP"] AND machine_name IS IN ["fi-kbicl-7560u2", "fi-icfl-gu2c", "fi-cfl-8109u", "fi-icl-u3", "fi-kbl-7560u"] AND ((testsuite_name = "IGT" AND test_name IS IN ["igt@kms_plane_lowres@pipe-b-tiling-none", "igt@kms_plane@pixel-format-pipe-b-planes", "igt@kms_plane@pixel-format-pipe-c-planes", "igt@kms_plane@pixel-format-pipe-a-planes", "igt@kms_vblank@pipe-a-accuracy-idle", "igt@gem_mmap_gtt@forked-big-copy", "igt@kms_plane_scaling@pipe-c-scaler-with-clipping-clamping", "igt@kms_plane_scaling@pipe-b-scaler-with-clipping-clamping", "igt@kms_flip@2x-blocking-wf_vblank", "igt@kms_flip@2x-flip-vs-modeset-vs-hang", "igt@kms_vblank@pipe-a-wait-idle", "igt@gem_mmap_gtt@big-copy-xy", "igt@kms_vblank@pipe-c-wait-forked-hang", "igt@kms_plane@pixel-format-pipe-c-planes-source-clamping", "igt@kms_rotation_crc@multiplane-rotation-cropping-top"])) AND ((testsuite_name = "IGT" AND status_name IS IN ["incomplete"]))

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_721/fi-cfl-guc/igt@kms_vblank@pipe-a-accuracy-idle.html

A CI Bug Log filter associated to this bug has been updated by :

Description: fi-cfl-guc,fi-kbl-7560u, fi-cfl-8109u, fi-icl-u2, fi-icl-u3: random tests - incomplete

Equivalent query: runconfig_tag IS IN ["DRM-TIP"] AND machine_name IS IN ["fi-icl-u2", "fi-cfl-guc", "fi-cfl-8109u", "fi-icl-u3", "fi-kbl-7560u"] AND ((testsuite_name = "IGT" AND test_name IS IN ["igt@kms_plane_lowres@pipe-b-tiling-none", "igt@kms_plane@pixel-format-pipe-b-planes", "igt@kms_plane@pixel-format-pipe-c-planes", "igt@kms_plane@pixel-format-pipe-a-planes", "igt@kms_vblank@pipe-a-accuracy-idle", "igt@gem_mmap_gtt@forked-big-copy", "igt@kms_plane_scaling@pipe-c-scaler-with-clipping-clamping", "igt@kms_plane_scaling@pipe-b-scaler-with-clipping-clamping", "igt@kms_flip@2x-blocking-wf_vblank", "igt@kms_flip@2x-flip-vs-modeset-vs-hang", "igt@kms_vblank@pipe-a-wait-idle", "igt@gem_mmap_gtt@big-copy-xy", "igt@kms_vblank@pipe-c-wait-forked-hang", "igt@kms_plane@pixel-format-pipe-c-planes-source-clamping", "igt@kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-draw-mmap-gtt", "igt@kms_rotation_crc@multiplane-rotation-cropping-top"])) AND ((testsuite_name = "IGT" AND status_name IS IN ["incomplete"]))

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_731/fi-cfl-8109u/igt@kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-draw-mmap-gtt.html

A CI Bug Log filter associated to this bug has been updated by :

Description: fi-cfl-guc,fi-kbl-7560u, fi-cfl-8109u, fi-icl-u2, fi-icl-u3: random tests - incomplete

Equivalent query: runconfig_tag IS IN ["DRM-TIP"] AND machine_name IS IN ["fi-icl-u2", "fi-cfl-guc", "fi-cfl-8109u", "fi-icl-u3", "fi-kbl-7560u"] AND ((testsuite_name = "IGT" AND test_name IS IN ["igt@kms_plane_lowres@pipe-b-tiling-none", "igt@kms_flip@flip-vs-rmfb-interruptible", "igt@kms_plane@pixel-format-pipe-b-planes", "igt@kms_plane@pixel-format-pipe-c-planes", "igt@kms_plane@pixel-format-pipe-a-planes", "igt@kms_vblank@pipe-a-accuracy-idle", "igt@gem_mmap_gtt@forked-big-copy", "igt@kms_plane_scaling@pipe-c-scaler-with-clipping-clamping", "igt@kms_plane_scaling@pipe-b-scaler-with-clipping-clamping", "igt@kms_flip@2x-blocking-wf_vblank", "igt@kms_flip@2x-flip-vs-modeset-vs-hang", "igt@kms_vblank@pipe-a-wait-idle", "igt@gem_mmap_gtt@big-copy-xy", "igt@kms_vblank@pipe-c-wait-forked-hang", "igt@kms_plane@pixel-format-pipe-c-planes-source-clamping", "igt@kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-draw-mmap-gtt", "igt@kms_rotation_crc@multiplane-rotation-cropping-top"])) AND ((testsuite_name = "IGT" AND status_name IS IN ["incomplete"]))

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_739/fi-cfl-8109u/igt@kms_flip@flip-vs-rmfb-interruptible.html

A CI Bug Log filter associated to this bug has been updated by :

Description: fi-cfl-guc,fi-kbl-7560u, fi-cfl-8109u, fi-icl-u2, fi-icl-u3: random tests - incomplete

Equivalent query: runconfig_tag IS IN ["DRM-TIP"] AND machine_name IS IN ["fi-icl-u2", "fi-cfl-guc", "fi-cfl-8109u", "fi-icl-u3", "fi-kbl-7560u"] AND ((testsuite_name = "IGT" AND test_name IS IN ["igt@kms_plane_lowres@pipe-b-tiling-none", "igt@kms_flip@flip-vs-rmfb-interruptible", "igt@kms_plane@pixel-format-pipe-b-planes", "igt@kms_plane@pixel-format-pipe-c-planes", "igt@kms_plane@pixel-format-pipe-a-planes", "igt@kms_vblank@pipe-a-accuracy-idle", "igt@gem_mmap_gtt@forked-big-copy", "igt@gem_exec_suspend@basic-s3", "igt@kms_plane_scaling@pipe-c-scaler-with-clipping-clamping", "igt@kms_plane_scaling@pipe-b-scaler-with-clipping-clamping", "igt@kms_flip@2x-blocking-wf_vblank", "igt@kms_flip@2x-flip-vs-modeset-vs-hang", "igt@kms_vblank@pipe-a-wait-idle", "igt@gem_mmap_gtt@big-copy-xy", "igt@kms_vblank@pipe-c-wait-forked-hang", "igt@kms_plane@pixel-format-pipe-c-planes-source-clamping", "igt@kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-draw-mmap-gtt", "igt@kms_rotation_crc@multiplane-rotation-cropping-top"])) AND ((testsuite_name = "IGT" AND status_name IS IN ["incomplete"]))

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_739/fi-icl-u2/igt@gem_exec_suspend@basic-s3.html

@adrinael Is this still valid?

Not valid anymore. Jenkins timeout no longer marks tests as incomplete.

Closing this issue as it's not valid anymore.

closed

The CI Bug Log issue associated to this bug has been archived.

New failures matching the above filters will not be associated to this bug anymore.

mentioned in commit b35bfa32

mentioned in commit c01d2733

mentioned in commit c5ca214c

mentioned in commit c5f54721

mentioned in commit c50e481d