[CI][RESUME] igt@kms_chamelium@dp-crc-multiple - fail - Chamelium RPC call failed: RPC failed at server.*:DP FSM failed

Submitted by Martin Peres @mupuf

Assigned to Intel GFX Bugs mailing list

Link to original bug (#110387)

Description

Starting subtest: dp-crc-multiple
(kms_chamelium:1203) igt_chamelium-CRITICAL: Test assertion failure function chamelium_rpc, file ../lib/igt_chamelium.c:306:
(kms_chamelium:1203) igt_chamelium-CRITICAL: Failed assertion: !chamelium->env.fault_occurred
(kms_chamelium:1203) igt_chamelium-CRITICAL: Chamelium RPC call failed: RPC failed at server. <class 'chameleond.devices.input_flow.InputFlowError'>:DP FSM failed

added CI feature: display/DP platform: CML platform: ICL platform: KBL priority::low severity::normal + 1 deleted label

💬 CI Bug Log said:

The CI Bug Log issue associated to this bug has been updated.

New filters associated

* ICL: igt@kms_chamelium@dp-crc-multiple - dmesg-fail - Chamelium RPC call failed: RPC failed at server.*:DP FSM failed

- https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_5891/re-icl-u/igt@kms_chamelium@dp-crc-multiple.html

💬 CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

{- ICL: igt@kms_chamelium@dp-crc-multiple - dmesg-fail - Chamelium RPC call failed: RPC failed at server.:DP FSM failed -}
{+ ICL: igt@kms_chamelium@dp-crc-(fast|multiple) - fail / dmesg-fail - Chamelium RPC call failed: RPC failed at server.:DP FSM failed +}

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_5960/re-icl-u/igt@kms_chamelium@dp-crc-fast.html

💬 Arkadiusz Hiler @ivyl said:

This is quite likely test's fault. When the DP receiver on the chamelium side needs its Finite State Machine handling due to mode/configuration changes it signals that to us by sending hot-plug events. Then we need to do DPMS cycle.

Current implementations spawns a thread monitoring for those events, but that seems to be prone to race conditions, and if we make XMLRPC call to chamelium before it is done we can get this error.

We have to re-implement FSM handling at the call site, so whenever we get this error we force the handling (or wait till the handling is done).

I have seen similar behavior on my local SKL too.

💬 CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

{- ICL: igt@kms_chamelium@dp-crc-(fast|multiple) - fail / dmesg-fail - Chamelium RPC call failed: RPC failed at server.:DP FSM failed -}
{+ CML ICL: igt@kms_chamelium@dp-crc-(fast|multiple|single) - fail / dmesg-fail - Chamelium RPC call failed: RPC failed at server.:DP FSM failed +}

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_276/fi-cml-u2/igt@kms_chamelium@dp-crc-single.html
  
• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_277/fi-cml-u2/igt@kms_chamelium@dp-crc-multiple.html

💬 James Ausmus said:

Hasn't been seen on ICL in 285 runs. Removing ICL from the affected platform list.

CML hasn't seen this in 123 runs, will keep this open for CML to monitor for re-occurence

💬 CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

{- CML ICL: igt@kms_chamelium@dp-crc-(fast|multiple|single) - fail / dmesg-fail - Chamelium RPC call failed: RPC failed at server.:DP FSM failed -}
{+ KBL CML ICL: igt@kms_chamelium@dp-crc-(fast|multiple|single) - fail / dmesg-fail - Chamelium RPC call failed: RPC failed at server.:DP FSM failed +}

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_369/fi-kbl-7500u/igt@kms_chamelium@dp-audio.html

A CI Bug Log filter associated to this bug has been updated:

{- KBL CML ICL: igt@kms_chamelium@dp-crc-(fast|multiple|single) - fail / dmesg-fail - Chamelium RPC call failed: RPC failed at server.:DP FSM failed -} {+ KBL CML ICL EHL: igt@kms_chamelium@dp-crc-(fast|multiple|single) - fail / dmesg-fail - Chamelium RPC call failed: RPC failed at server.:DP FSM failed +}

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_423/fi-cml-u2/igt@kms_chamelium@dp-frame-dump.html
• https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7827/re-ehl-1/igt@kms_color_chamelium@pipe-c-ctm-0-75.html

added platform: EHL label

A CI Bug Log filter associated to this bug has been updated:

{- KBL CML ICL EHL: igt@kms_chamelium@dp-crc-(fast|multiple|single) - fail / dmesg-fail - Chamelium RPC call failed: RPC failed at server.:DP FSM failed -} {+ KBL CML ICL EHL: igt@kms_chamelium@ - fail / dmesg-fail - Chamelium RPC call failed: RPC failed at server.*:DP FSM failed +}

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_428/fi-cml-u2/igt@kms_color_chamelium@pipe-a-ctm-red-to-blue.html
• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_428/fi-icl-u2/igt@kms_color_chamelium@pipe-c-ctm-0-25.html
• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_428/fi-kbl-7500u/igt@kms_color_chamelium@pipe-a-ctm-0-5.html
• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_429/fi-kbl-7500u/igt@kms_color_chamelium@pipe-a-ctm-red-to-blue.html

A CI Bug Log filter associated to this bug has been updated by Lakshmi Vudum:

Description: KBL CML ICL EHL TGL: igt@kms_chamelium@* - fail / dmesg-fail - Chamelium RPC call failed: RPC failed at server.*:DP FSM failed

Equivalent query: runconfig_tag IS IN ["DRM-TIP"] AND (machine_name IS IN ["fi-kbl-soraka", "shard-kbl3", "fi-icl-u2", "shard-iclb", "shard-iclb1", "shard-iclb2", "shard-iclb3", "shard-iclb4", "shard-iclb5", "shard-iclb6", "fi-icl-u3", "shard-iclb7", "fi-icl-y", "shard-kbl4", "shard-iclb8", "fi-icl-guc", "fi-icl-dsi", "re-icl-u", "shard-kbl7", "fi-cml-u", "re-cml-u", "fi-cml-u2", "fi-icl-u4", "fi-tgl-u", "fi-cml-h", "fi-cml-s", "re-tgl-u", "fi-tgl-u2", "shard-tglb1", "shard-tglb2", "shard-tglb3", "shard-tglb4", "shard-tglb5", "shard-tglb6", "fi-kbl-7500u", "re-tgl1-display", "re-tgl2-display", "shard-kbl6", "shard-tglb7", "shard-tglb8", "shard-kbl1", "fi-kbl-7560u", "fi-tgl-y", "re-tgl-y", "shard-tglb0", "shard-tglb9", "fi-kbl-r", "fi-icl-1065g7", "fi-kbl-7567u", "fi-tgl-guc", "fi-ehl-1", "re-ehl-1", "re-tgl-cham", "fi-tgl-dsi", "re-tgl-u2", "ba-tgl-1", "re-ehl-cham", "shard-kbl2", "shard-kbl5", "shard-kbl", "fi-kbl-guc", "fi-kbl-x1275", "fi-kbl-8809g", "fi-icl-u"] OR machine_tag IS IN ["EHL", "KBL", "ICL", "CML", "TGL", "EHL", "KBL"]) AND ((testsuite_name = "IGT" AND test_name IS IN ["igt@kms_chamelium@dp-crc-fast", "igt@kms_color_chamelium@pipe-b-ctm-max", "igt@kms_color_chamelium@pipe-b-ctm-negative", "igt@kms_color_chamelium@pipe-b-gamma", "igt@kms_color_chamelium@pipe-c-ctm-0-75", "igt@kms_color_chamelium@pipe-a-ctm-max", "igt@kms_color_chamelium@pipe-a-gamma", "igt@kms_color_chamelium@pipe-d-ctm-0-25", "igt@kms_color_chamelium@pipe-d-ctm-red-to-blue", "igt@kms_color_chamelium@pipe-d-ctm-green-to-red", "igt@kms_color_chamelium@pipe-d-degamma", "igt@kms_color_chamelium@pipe-c-ctm-0-5", "igt@kms_color_chamelium@pipe-a-degamma", "igt@kms_color_chamelium@pipe-a-ctm-blue-to-red", "igt@kms_color_chamelium@pipe-a-ctm-limited-range", "igt@kms_color_chamelium@pipe-b-ctm-limited-range", "igt@kms_color_chamelium@pipe-b-ctm-0-25", "igt@kms_color_chamelium@pipe-a-ctm-green-to-red", "igt@kms_color_chamelium@pipe-invalid-degamma-lut-sizes", "igt@kms_color_chamelium@pipe-c-ctm-blue-to-red", "igt@kms_color_chamelium@pipe-c-ctm-limited-range", "igt@kms_color_chamelium@pipe-b-degamma", "igt@kms_color_chamelium@pipe-d-ctm-max", "igt@kms_color_chamelium@pipe-b-ctm-green-to-red", "igt@kms_color_chamelium@pipe-a-ctm-0-5", "igt@kms_color_chamelium@pipe-invalid-ctm-matrix-sizes", "igt@kms_color_chamelium@pipe-c-gamma", "igt@kms_color_chamelium@pipe-d-gamma", "igt@kms_chamelium@dp-frame-dump", "igt@kms_color_chamelium@pipe-a-ctm-negative", "igt@kms_color_chamelium@pipe-c-ctm-negative", "igt@kms_color_chamelium@pipe-a-ctm-0-75", "igt@kms_color_chamelium@pipe-b-ctm-red-to-blue", "igt@kms_color_chamelium@pipe-d-ctm-0-75", "igt@kms_color_chamelium@pipe-d-ctm-limited-range", "igt@kms_color_chamelium@pipe-d-ctm-negative", "igt@kms_color_chamelium@pipe-b-ctm-0-75", "igt@kms_color_chamelium@pipe-c-ctm-red-to-blue", "igt@kms_color_chamelium@pipe-c-ctm-green-to-red", "igt@kms_color_chamelium@pipe-c-ctm-0-25", "igt@kms_color_chamelium@pipe-c-degamma", "igt@kms_color_chamelium@pipe-a-ctm-red-to-blue", "igt@kms_color_chamelium@pipe-d-ctm-0-5", "igt@kms_color_chamelium@pipe-c-ctm-max", "igt@kms_color_chamelium@pipe-b-ctm-blue-to-red", "igt@kms_color_chamelium@pipe-d-ctm-blue-to-red", "igt@kms_color_chamelium@pipe-a-ctm-0-25", "igt@kms_color_chamelium@pipe-b-ctm-0-5", "igt@kms_chamelium@dp-crc-multiple", "igt@kms_color_chamelium@pipe-invalid-gamma-lut-sizes", "igt@kms_chamelium@dp-audio", "igt@kms_chamelium@dp-crc-single"])) AND ((testsuite_name = "IGT" AND status_name IS IN ["fail", "dmesg-fail"])) AND stderr ~= 'Chamelium RPC call failed: RPC failed at server.*:DP FSM failed'

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8076/re-tgl-cham/igt@kms_chamelium@dp-crc-single.html
• https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8076/re-tgl-cham/igt@kms_chamelium@dp-crc-multiple.html
• https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8076/re-tgl-cham/igt@kms_chamelium@dp-frame-dump.html

added platform: TGL label

mentioned in commit igt-gpu-tools@c8dc1fd9

mentioned in commit igt-gpu-tools@0c5b0e37

Issue is reproducing on CML,KBL,ICL

• https://intel-gfx-ci.01.org/tree/drm-tip/IGT_5833/fi-kbl-7500u/igt@kms_chamelium@dp-crc-fast.html
• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_702/fi-cml-u2/igt@kms_color_chamelium@pipe-b-ctm-0-5.html
• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_702/fi-icl-u2/igt@kms_color_chamelium@pipe-c-ctm-0-5.html

The CI Bug Log issue associated to this bug has been updated by .

New filters associated

• KBL : IGT: igt@kms_chamelium@dp-crc-fast - dmesg fail - no useful logs
  • https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5102/fi-kbl-7500u/igt@kms_chamelium@dp-crc-fast.html
  • https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_18822/fi-kbl-7500u/igt@kms_chamelium@dp-crc-fast.html
  • https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9217/fi-kbl-7500u/igt@kms_chamelium@dp-crc-fast.html
  • https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9190/fi-kbl-7500u/igt@kms_chamelium@dp-crc-fast.html

A CI Bug Log filter associated to this bug has been updated by :

Description: KBL CFL : IGT: igt@kms_chamelium@dp-crc-fast - dmesg fail - no useful logs

Equivalent query: runconfig_tag IS IN ["DRM-TIP"] AND (machine_name IS IN ["fi-kbl-7567u", "fi-kbl-soraka", "shard-kbl3", "shard-kbl2", "fi-kbl-guc", "fi-kbl-7500u", "shard-kbl5", "shard-kbl6", "fi-kbl-x1275", "fi-cfl-8109u", "shard-kbl", "fi-kbl-8809g", "shard-kbl4", "shard-kbl1", "fi-kbl-7560u", "shard-kbl7", "fi-kbl-r"] OR machine_tag IS IN ["KBL"]) AND ((testsuite_name = "IGT" AND test_name IS IN ["igt@kms_chamelium@dp-crc-fast", "igt@debugfs_test@read_all_entries", "igt@debugfs_test@read_all_entries_display_on", "igt@gem_exec_suspend@basic-s0", "igt@debugfs_test@read_all_entries_display_off"])) AND ((testsuite_name = "IGT" AND status_name IS IN ["dmesg-warn"]))

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9245/fi-cfl-8109u/igt@debugfs_test@read_all_entries.html
• https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9245/fi-cfl-8109u/igt@gem_exec_suspend@basic-s0.html

The CI Bug Log issue associated to this bug has been updated by .

New filters associated

• KBL igt@perf_pmu@module-unload | igt@gem_sync@basic-store-all- dmesg-warn
  • https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9282/shard-iclb7/igt@kms_dp_aux_dev.html
  • https://intel-gfx-ci.01.org/tree/drm-tip/IGT_5838/shard-iclb3/igt@gem_sync@basic-store-all.html
  • https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_701/fi-kbl-7500u/igt@perf_pmu@module-unload.html
  • https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9295/shard-iclb7/igt@kms_dp_aux_dev.html
  • https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_18873/shard-hsw6/igt@perf_pmu@module-unload.html
  • https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9296/shard-hsw6/igt@perf_pmu@module-unload.html
  • https://intel-gfx-ci.01.org/tree/drm-tip/IGT_5845/shard-iclb7/igt@kms_dp_aux_dev.html
  • https://intel-gfx-ci.01.org/tree/drm-tip/Trybot_7210/shard-iclb7/igt@kms_dp_aux_dev.html
  • https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9321/shard-iclb7/igt@kms_dp_aux_dev.html

mentioned in commit igt-gpu-tools@35526957

Issue is reproducing on KBL with IGT_6193 , drmtip_897.

Log: igt_kms_chamelium_dp-hpd-storm-disable_on_shard-kbl_CI_DRM_10560_Test_details.html

Issue is not observed on TGL, CML, EHL,ICL platforms for more than a month respectively. Hence removing above mentioned platform tags from #262 (closed)