[BAT][CI][GLK only] igt@* - incomplete - timeout/system hang
Submitted by Marta Löfstedt @marta
Assigned to Intel GFX Bugs mailing list
Link to original bug (#103359)
Description
CI_DRM_3265 GLK-shards igt@gem_softpin@noreloc-S3
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3265/shard-glkb3/igt@gem_softpin@noreloc-S3.html
Note bootlog start at:
<7>[ 548.719874] [drm:edp_panel_vdd_off_sync [i915]] Turning eDP port A VDD off
i.e there has been at least one shard of tests run before this on on this boot.
last dmesg:
<6>[ 1005.896525] PM: suspend entry (deep)
run.log:
[13/73] skip: 5, pass: 7, dmesg-warn: 1 /
FATAL: command execution failed
java.io.EOFException
there is:
[ 2603.510504] vblank not available on crtc 0, ret=-22
[ 2603.510562] ------------[ cut here ]------------
[ 2603.510585] WARNING: CPU: 2 PID: 5328 at drivers/gpu/drm/drm_vblank.c:1042 drm_wait_one_vblank+0x160/0x1b0
in this dmesg.
Blocking
Activity
added CI feature: GEM feature: display/Other platform: GLK priority::medium severity::normal + 1 deleted label
@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3267/shard-glkb4/igt@kms_frontbuffer_tracking@psr-shrfb-scaledprimary.html
The backtrace has been cut-off:
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3267/shard-glkb4/dmesg-1508445725_Oops_1.log
last dmesg:
<7>[ 2884.501987] [IGT] kms_plane_multiple: exiting, ret=77
<7>[ 2884.658294] [IGT] kms_frontbuffer_tracking: executing
then stray.
run.log:
[22/72] skip: 5, pass: 15, fail: 2 -
FATAL: command execution failed
java.io.EOFException
dmesg start at:
<7>[ 2753.040777] [IGT] kms_frontbuffer_tracking: executing
so other shards have been run on this machine before.@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3268/shard-glkb1/igt@gem_eio@in-flight-contexts.html
last demesg followed by stray:
<7>[ 462.664336] [drm:gen8_init_common_ring [i915]] Execlists enabled for vcs0
<7>[ 462.664534] [drm:gen8_init_common_ring [i915]] Execlists enabled for vecs0
<4>[ 462.689948] Setting dangerous option reset - tainting kernel
otherwise no ERROR in dmesg
run-log:
[40/72] skip: 18, pass: 16, fail: 6 |
FATAL: command execution failed
java.io.EOFException@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3275/shard-glkb3/igt@gem_ppgtt@blt-vs-render-ctx0.html
last dmesg:
<7>[ 1175.713017] [IGT] gem_ppgtt: executing
<7>[ 1175.737657] [IGT] gem_ppgtt: starting subtest blt-vs-render-ctx0
<6>[ 1256.231064] perf: interrupt took too long (2509 > 2500), lowering kernel.perf_event_max_sample_rate to 79000
run.log:
[33/73] skip: 11, pass: 19, fail: 3 /
FATAL: command execution failed
java.io.EOFException@martasaid:CI_DRM_3286 shard-glkb1 igt@gem_cpu_reloc@full
First dmesg:
<5>[ 1648.677701] owatch: Using watchdog device /dev/watchdog0
Last dmesg:
<7>[ 2012.063887] [IGT] gem_cpu_reloc: starting subtest full
run.log:
[33/73] skip: 11, pass: 20, fail: 2 /
FATAL: command execution failed
java.io.EOFException
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638)
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3286/shard-glkb1/igt@gem_cpu_reloc@full.html@martasaid:CI_DRM_3286 shard-glkb2 igt@gem_tiled_partial_pwrite_pread@reads
First dmesg:
<5>[ 806.330960] owatch: Using watchdog device /dev/watchdog0
Last dmesg:
<7>[ 997.929771] [IGT] gem_tiled_partial_pwrite_pread: executing
[00/73] |
FATAL: command execution failed
java.io.EOFException
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3286/shard-glkb2/igt@gem_tiled_partial_pwrite_pread@reads.html@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3289/shard-glkb1/igt@gem_ringfill@basic-default-hang.html
First dmesg:
<5>[ 3379.148765] owatch: Using watchdog device /dev/watchdog0
Last dmesg:
<7>[ 3694.039373] [IGT] gem_ringfill: executing
Followed by "stay".
runtimes log:
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3289/shard-glkb1/runtimes31.txt
=> ~5 minutes
Note dmesg timestamps are not wall time, the runtime should be. Also, Arek just fixed this feature with runtime just for the tests in each shard, thanks Arek!
run.log:
[11/72] skip: 3, pass: 8 \
FATAL: command execution failed
java.io.EOFException
From this I draw the conclusion that this was a system hang.@martasaid:I will also file incomplete timeout/system hang on fi-glk-dsi on this bug. However, note bug 103514 which is a full scenario where we start with wedged GPU and then end up in incomplete. This is not the case for this: CI_DRM_3305 fi-glk-dsi igt@gem_ctx_switch@basic-default-heavy
last dmesg:
<7>[ 75.283189] [IGT] gem_ctx_switch: executing
Followed by stry.
run.log:
[031/289] skip: 9, pass: 22 \
FATAL: command execution failed
java.io.EOFException
CI_IGT_test runtime 175 seconds
So, this looks like system hang.
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3305/fi-glk-dsi/igt@gem_ctx_switch@basic-default-heavy.html@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3308/shard-glkb4/igt@kms_frontbuffer_tracking@fbc-1p-primscrn-pri-indfb-draw-blt.html
First dmesg:
<5>[ 1593.174703] owatch: Using watchdog device /dev/watchdog0
<5>[ 1593.174783] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 1593.177277] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
Last dmesg:
<7>[ 1979.031808] [drm:drm_helper_hpd_irq_event] [CONNECTOR:89:DP-1] status updated from disconnected to disconnected
<7>[ 1979.183337] [IGT] kms_frontbuffer_tracking: executing
Followed by stray.
run.log:
[55/73] skip: 12, pass: 38, dmesg-warn: 1, fail: 4 \
FATAL: command execution failed
java.io.EOFException
...
Completed CI_IGT_test CI_DRM_3308@shard-glkb4 : FAILURE
CI_IGT_test runtime 463 seconds@martasaid:CI_DRM_3312 fi-glk-dsi igt@gem_exec_flush@basic-wb-set-default
dmesg:
<5>[ 43.114273] owatch: Using watchdog device /dev/watchdog0
<5>[ 43.114505] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 43.116098] owatch: timeout for /dev/watchdog0 set to 100 (requested 100)
...
<7>[ 229.819324] [IGT] gem_exec_flush: executing
run.log
[073/289] skip: 16, pass: 57 /
FATAL: command execution failed
java.io.EOFException
....
Completed CI_IGT_test CI_DRM_3312@fi-glk-dsi : FAILURE
CI_IGT_test runtime 286 seconds
looks like system hang.
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3312/fi-glk-dsi/igt@gem_exec_flush@basic-wb-set-default.html@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3328/shard-glkb3/igt@gem_exec_blt@cold-max.html
dmesg:
<5>[ 1760.361917] owatch: Using watchdog device /dev/watchdog0
<5>[ 1760.361996] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 1760.364687] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
no obvious ERROR
...
<7>[ 1985.345189] [IGT] gem_exec_params: executing
<7>[ 1985.372299] [IGT] gem_exec_params: starting subtest cliprects_ptr-dirt
<7>[ 1985.372502] [IGT] gem_exec_params: exiting, ret=0
Followed by stray.
run.log:
running: igt/kms_flip/vblank-vs-modeset-suspend-interruptible
[34/73] skip: 16, pass: 16, fail: 2 -
FATAL: command execution failed
java.io.EOFException
...
Finished: FAILURE
Completed CI_IGT_test CI_DRM_3328@shard-glkb3 : FAILURE
CI_IGT_test runtime 515 seconds@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3326/shard-glkb3/igt@gem_softpin@softpin.html
dmesg:
<5>[ 3641.153567] owatch: Using watchdog device /dev/watchdog0
<5>[ 3641.153647] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 3641.156151] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
no ERRORs
...
<7>[ 3974.363963] [IGT] gem_readwrite: exiting, ret=0
<7>[ 3974.540413] [IGT] gem_softpin: executing
<7>[ 3974.555885] [IGT] gem_softpin: starting subtest softpin
run.log:
running: igt/gem_softpin/softpin
[60/72] skip: 15, pass: 43, fail: 2 |
FATAL: command execution failed
...
Finished: FAILURE
Completed CI_IGT_test CI_DRM_3326@shard-glkb3 : FAILURE
CI_IGT_test runtime 514 seconds@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3334/shard-glkb1/igt@perf@polling.html
dmesg:
<5>[ 2456.779388] owatch: Using watchdog device /dev/watchdog0
<5>[ 2456.779456] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 2456.780498] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
<7>[ 2705.896434] [drm:intel_power_well_disable [i915]] disabling power well 2
<7>[ 2705.896455] [drm:drm_helper_hpd_irq_event] [CONNECTOR:89:DP-1] status updated from disconnected to disconnected
<7>[ 2706.025517] [IGT] perf: executing
run.log
[04/72] skip: 1, pass: 2, fail: 1 |
FATAL: command execution failed
java.io.EOFException
...
Completed CI_IGT_test CI_DRM_3334@shard-glkb1 : FAILURE
CI_IGT_test runtime 339 seconds@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3339/shard-glkb2/igt@kms_frontbuffer_tracking@fbc-suspend.html
dmesg:
<5>[ 470.310793] owatch: Using watchdog device /dev/watchdog0
<5>[ 470.310870] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 470.312856] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
<7>[ 753.111566] [drm:intel_power_well_disable [i915]] disabling DC off
<7>[ 753.111665] [drm:gen9_enable_dc5 [i915]] Enabling DC5
<7>[ 753.111759] [drm:gen9_set_dc_state [i915]] Setting DC state from 00 to 01
run.log:
[30/73] skip: 12, pass: 15, fail: 3 -
FATAL: command execution failed
java.io.EOFException
...
Completed CI_IGT_test CI_DRM_3339@shard-glkb2 : FAILURE
CI_IGT_test runtime 387 seconds@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3339/shard-glkb5/igt@kms_flip@vblank-vs-modeset-suspend-interruptible.html
above is just weird, impossible to decide which pstore that belongs to this run, timestamstes indicates runtimes over 24 hours.@martasaid:this is from same run as previous:
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3339/shard-glkb5/igt@kms_busy@basic-flip-b.html
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3339/shard-glkb5/igt@kms_plane@plane-panning-bottom-right-suspend-pipe-a-planes.html@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3340/shard-glkb5/igt@pm_rpm@system-suspend.html
dmesg:
<5>[ 1627.977850] owatch: Using watchdog device /dev/watchdog0
<5>[ 1627.977929] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 1627.979299] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
<7>[ 1713.425491] [drm:hsw_power_well_disable [i915]] power well 1 forced on (bios:1 driver:0 kvmr:0 debug:0)
<7>[ 1713.425583] [drm:bxt_enable_dc9 [i915]] Enabling DC9
<7>[ 1713.425623] [drm:gen9_set_dc_state [i915]] Setting DC state from 00 to 08
<7>[ 1713.425686] [drm:intel_runtime_suspend [i915]] Device suspended
run.log
[25/72] skip: 6, pass: 19 /
FATAL: command execution failed
java.io.EOFException
...
Completed CI_IGT_test CI_DRM_3340@shard-glkb5 : FAILURE
CI_IGT_test runtime 441 seconds@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3341/shard-glkb5/igt@gem_workarounds@suspend-resume-fd.html
dmesg:
<5>[ 25.911268] owatch: Using watchdog device /dev/watchdog0
<5>[ 25.911583] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 25.912970] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
<7>[ 65.759686] [IGT] gem_workarounds: executing
<7>[ 65.782123] [IGT] gem_workarounds: starting subtest suspend-resume-fd
<6>[ 65.828745] PM: suspend entry (deep)
run.log:
[11/72] skip: 3, pass: 7, fail: 1 \
FATAL: command execution failed
java.io.EOFException
...
Completed CI_IGT_test CI_DRM_3341@shard-glkb5 : FAILURE
CI_IGT_test runtime 548 seconds@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3342/shard-glkb5/igt@drv_suspend@fence-restore-untiled.html
dmesg:
<5>[ 27.357629] owatch: Using watchdog device /dev/watchdog0
<5>[ 27.357971] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 27.359511] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
<7>[ 147.392151] [IGT] gem_exec_blt: exiting, ret=0
<7>[ 147.504680] [IGT] drv_suspend: executing
<7>[ 147.529732] [IGT] drv_suspend: starting subtest fence-restore-untiled
run.log
[19/72] skip: 8, pass: 11 \
Build timed out (after 17 minutes). Marking the build as aborted.
FATAL: command execution failed
java.io.IOException
...
Completed CI_IGT_test CI_DRM_3342@shard-glkb5 : FAILURE
CI_IGT_test runtime 1028 seconds
Note: Jenkins timeout after 17 minutes. Last dmesg is at 147 seconds!@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3343/shard-glkb5/igt@kms_cursor_crc@cursor-256x256-suspend.html
dmesg:
<5>[ 469.226040] owatch: Using watchdog device /dev/watchdog0
<5>[ 469.226117] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 469.227135] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
<7>[ 722.630630] [IGT] kms_cursor_crc: starting subtest cursor-256x256-suspend
<7>[ 722.630905] [drm:drm_mode_addfb2] [FB:98]
<7>[ 722.645396] [drm:drm_mode_setcrtc] [CRTC:42:pipe A]
<7>[ 722.645453] [drm:drm_mode_setcrtc] [CONNECTOR:76:eDP-1]
<7>[ 722.661151] [drm:drm_mode_setcrtc] [CRTC:58:pipe B]
<7>[ 722.661687] [drm:drm_mode_setcrtc] [CRTC:74:pipe C]
Folloed by "stray".
run.log:
[66/72] skip: 24, pass: 37, fail: 5 -
FATAL: command execution failed
java.io.EOFException
...
Completed CI_IGT_test CI_DRM_3343@shard-glkb5 : FAILURE
CI_IGT_test runtime 709 seconds@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3344/shard-glkb6/igt@kms_atomic_transition@plane-all-transition-nonblocking-fencing.html
dmesg:
<5>[ 1949.061995] owatch: Using watchdog device /dev/watchdog0
<5>[ 1949.062289] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 1949.062784] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
<7>[ 2230.806594] [drm:intel_power_well_disable [i915]] disabling power well 2
<7>[ 2230.807064] [drm:drm_helper_hpd_irq_event] [CONNECTOR:89:DP-1] status updated from disconnected to disconnected
<7>[ 2231.121060] [IGT] kms_atomic_transition: executing
Followed by "stray"
run.log is weird doesn't show that any tests has been executed:
running: igt/drv_suspend/forcewake-hibernate
[00/72] |
FATAL: command execution failed
java.io.EOFException
...
Finished: FAILURE
Completed CI_IGT_test CI_DRM_3344@shard-glkb6 : FAILURE
CI_IGT_test runtime 380 seconds
So, it looks like Jenkins lost connection and rebooted the machine. Or, 380 seconds is suspiciously similar to 370s watchdog timeout set by owatch. So, if@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3344/shard-glkb5/igt@kms_plane@plane-panning-bottom-right-suspend-pipe-b-planes.html
dmesg:
<5>[ 25.149724] owatch: Using watchdog device /dev/watchdog0
<5>[ 25.149958] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 25.151533] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
<7>[ 558.126780] [drm:verify_connector_state.isra.74 [i915]] [CONNECTOR:76:eDP-1]
<7>[ 558.126860] [drm:intel_atomic_commit_tail [i915]] [CRTC:58:pipe B]
<7>[ 558.126977] [drm:verify_single_dpll_state.isra.75 [i915]] PORT PLL A
run.log:
[72/73] skip: 18, pass: 53, fail: 1 |
FATAL: command execution failed
java.io.EOFException
...
Completed CI_IGT_test CI_DRM_3344@shard-glkb5 : FAILURE
CI_IGT_test runtime 848 seconds@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3346/shard-glkb5/igt@gem_exec_suspend@basic-s3.html
dmesg:
<5>[ 27.257860] owatch: Using watchdog device /dev/watchdog0
<5>[ 27.258450] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 27.262545] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
<7>[ 75.294604] [IGT] gem_exec_suspend: executing
<4>[ 75.312907] Setting dangerous option reset - tainting kernel
<7>[ 75.315159] [IGT] gem_exec_suspend: starting subtest basic-S3
Followed by "stray"
run.log:
[08/72] skip: 3, pass: 5 |
FATAL: command execution failed
java.io.EOFException
...
Finished: FAILURE
Completed CI_IGT_test CI_DRM_3346@shard-glkb5 : FAILURE
CI_IGT_test runtime 550 seconds@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3347/shard-glkb5/igt@gem_workarounds@suspend-resume.html
dmesg:
<5>[ 390.922353] owatch: Using watchdog device /dev/watchdog0
<5>[ 390.922432] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 390.923494] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
<7>[ 586.552858] [IGT] gem_workarounds: executing
<7>[ 586.576935] [IGT] gem_workarounds: starting subtest suspend-resume
<6>[ 586.622368] PM: suspend entry (deep)
run.log:
[26/73] skip: 6, pass: 18, fail: 2 -
FATAL: command execution failed
java.io.EOFException
...
Finished: FAILURE
Completed CI_IGT_test CI_DRM_3347@shard-glkb5 : FAILURE
CI_IGT_test runtime 485 seconds@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3348/shard-glkb5/igt@kms_cursor_crc@cursor-64x64-suspend.html
dmesg:
<5>[ 23.777582] owatch: Using watchdog device /dev/watchdog0
<5>[ 23.777805] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 23.779501] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
<7>[ 422.722911] [drm:drm_mode_setcrtc] [CONNECTOR:76:eDP-1]
<7>[ 422.734843] [drm:drm_mode_setcrtc] [CRTC:58:pipe B]
<7>[ 422.735662] [drm:drm_mode_setcrtc] [CRTC:74:pipe C]
run.log:
[64/73] skip: 20, pass: 39, fail: 5 |
FATAL: command execution failed
java.io.EOFException
...
Finished: FAILURE
Completed CI_IGT_test CI_DRM_3348@shard-glkb5 : FAILURE
CI_IGT_test runtime 847 seconds
Rebooting shard-glkb5@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3349/shard-glkb6/igt@gem_exec_create@madvise.html
dmesg:
<5>[ 647.132764] owatch: Using watchdog device /dev/watchdog0
<5>[ 647.132840] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 647.133861] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
<7>[ 875.417682] [drm:drm_helper_probe_single_connector_modes] [CONNECTOR:89:DP-1] disconnected
<7>[ 875.419898] [IGT] kms_color: exiting, ret=77
<7>[ 875.612360] [IGT] gem_exec_create: executing
run.log
[16/72] skip: 7, pass: 8, fail: 1 |
FATAL: command execution failed
java.io.EOFException
...
Completed CI_IGT_test CI_DRM_3349@shard-glkb6 : FAILURE
CI_IGT_test runtime 342 seconds
Rebooting shard-glkb6@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3350/shard-glkb5/igt@kms_flip@vblank-vs-dpms-suspend-interruptible.html
<5>[ 685.382002] owatch: Using watchdog device /dev/watchdog0
<5>[ 685.382082] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 685.383453] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
<7>[ 956.156049] [drm:gen9_enable_dc5 [i915]] Enabling DC5
<7>[ 956.156118] [drm:gen9_set_dc_state [i915]] Setting DC state from 00 to 01
<7>[ 956.157209] [drm:intel_power_well_disable [i915]] disabling always-on
run.log:
[29/73] skip: 9, pass: 18, fail: 2 /
Build timed out (after 17 minutes). Marking the build as aborted.
FATAL: command execution failed
java.io.IOException
...
Completed CI_IGT_test CI_DRM_3350@shard-glkb5 : FAILURE
CI_IGT_test runtime 1021 seconds
Rebooting shard-glkb5@martasaid:<5>[ 36.684057] owatch: Using watchdog device /dev/watchdog0
<5>[ 36.684332] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 36.686046] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
<7>[ 124.274454] [drm:intel_enable_ddi [i915]] Panel doesn't support DRRS
<7>[ 124.291308] [drm:verify_connector_state.isra.74 [i915]] [CONNECTOR:76:eDP-1]
<7>[ 124.291366] [drm:intel_atomic_commit_tail [i915]] [CRTC:74:pipe C]
<7>[ 124.291479] [drm:verify_single_dpll_state.isra.75 [i915]] PORT PLL A
run.log:
[08/73] skip: 1, pass: 6, fail: 1 |
FATAL: command execution failed
java.io.EOFException
...
Completed CI_IGT_test CI_DRM_3352@shard-glkb5 : FAILURE
CI_IGT_test runtime 557 seconds
Rebooting shard-glkb5@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3354/shard-glkb5/igt@drv_suspend@sysfs-reader.html
dmesg:
<5>[ 369.207913] owatch: Using watchdog device /dev/watchdog0
<5>[ 369.207980] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 369.209059] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
<7>[ 620.578814] [drm:intel_backlight_device_get_brightness [i915]] get backlight PWM = 96000
<7>[ 620.586526] [drm:intel_backlight_device_get_brightness [i915]] get backlight PWM = 96000
<6>[ 620.608421] PM: suspend entry (deep)
run.log:
running: igt/drv_suspend/sysfs-reader
[39/73] skip: 16, pass: 21, fail: 2 <br> FATAL: command execution failed
java.io.EOFException
...
Completed CI_IGT_test CI_DRM_3354@shard-glkb5 : FAILURE
CI_IGT_test runtime 508 seconds
Rebooting shard-glkb5@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3355/shard-glkb5/igt@kms_flip@vblank-vs-dpms-suspend.html
dmesg:
<5>[ 26.618706] owatch: Using watchdog device /dev/watchdog0
<5>[ 26.618929] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 26.621515] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
<7>[ 400.814160] [drm:gen9_set_dc_state [i915]] Setting DC state from 00 to 01
<7>[ 400.814405] [drm:intel_dp_hpd_pulse [i915]] ignoring long hpd on eDP port A
<7>[ 400.815308] [drm:intel_power_well_disable [i915]] disabling always-on
run.log:
running: igt/kms_flip/vblank-vs-dpms-suspend
[32/73] skip: 12, pass: 19, fail: 1 |
FATAL: command execution failed
java.io.EOFException
...
Completed CI_IGT_test CI_DRM_3355@shard-glkb5 : FAILURE
CI_IGT_test runtime 847 seconds
Rebooting shard-glkb5@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3357/fi-glk-dsi/igt@gem_sync@basic-each.html
@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3360/shard-glkb5/igt@pm_rpm@system-suspend-execbuf.html
dmesg:
<5>[ 30.844398] owatch: Using watchdog device /dev/watchdog0
<5>[ 30.844728] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 30.846748] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
<7>[ 108.387979] [drm:gen9_set_dc_state [i915]] Setting DC state from 00 to 08
<7>[ 108.388079] [drm:intel_runtime_suspend [i915]] Device suspended
<6>[ 108.552651] PM: suspend entry (deep)
run.log
running: igt/pm_rpm/system-suspend-execbuf
[20/73] skip: 5, pass: 14, fail: 1 |
FATAL: command execution failed
java.io.EOFException
...
Completed CI_IGT_test CI_DRM_3360/shard-glkb5/21 : FAILURE
CI_IGT_test runtime 550 seconds
Rebooting shard-glkb5@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/IGT_3988/shard-glkb5/igt@drv_suspend@forcewake.html
dmesg:
<5>[ 25.000516] owatch: Using watchdog device /dev/watchdog0
<5>[ 25.000738] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 25.002445] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
<7>[ 164.159097] [IGT] drv_suspend: executing
<7>[ 164.180418] [IGT] drv_suspend: starting subtest forcewake
<6>[ 164.229837] PM: suspend entry (deep)
run.log:
running: igt/drv_suspend/forcewake
[28/73] skip: 9, pass: 18, fail: 1 |
FATAL: command execution failed
java.io.EOFException
...
Completed CI_IGT_test CI_DRM_3353@shard-glkb5 : FAILURE
CI_IGT_test runtime 546 seconds
Rebooting shard-glkb5@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/IGT_3990/shard-glkb5/igt@pm_rpm@system-suspend-modeset.html
dmesg:
<5>[ 26.165090] owatch: Using watchdog device /dev/watchdog0
<5>[ 26.165660] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 26.167442] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
<7>[ 210.371312] [drm:gen9_set_dc_state [i915]] Setting DC state from 00 to 08
<7>[ 210.371447] [drm:intel_runtime_suspend [i915]] Device suspended
<6>[ 210.553751] PM: suspend entry (deep)
run.log:
running: igt/pm_rpm/system-suspend-modeset
[31/73] skip: 12, pass: 19 \
Build timed out (after 17 minutes). Marking the build as aborted.
FATAL: command execution failed
java.io.IOException@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/IGT_3993/fi-glk-dsi/igt@gem_ctx_basic.html
dmesg:
<5>[ 38.501306] owatch: Using watchdog device /dev/watchdog0
<5>[ 38.501789] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 38.503219] owatch: timeout for /dev/watchdog0 set to 100 (requested 100)
...
<7>[ 69.732772] [IGT] gem_cs_tlb: executing
<7>[ 69.760911] [IGT] gem_cs_tlb: starting subtest basic-default
<7>[ 74.449651] [IGT] gem_cs_tlb: exiting, ret=0
Followed by "stray"
run.log:
running: igt/gem_ctx_basic
[024/289] skip: 9, pass: 15 |
FATAL: command execution failed
java.io.EOFException
...
Finished: FAILURE
Completed CI_IGT_test CI_DRM_3364/fi-glk-dsi/0 : FAILURE
CI_IGT_test runtime 549 seconds
Rebooting fi-glk-dsi@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3364/shard-glkb2/igt@kms_flip@blt-wf_vblank-vs-dpms.html
dmesg:
<5>[ 1956.055800] owatch: Using watchdog device /dev/watchdog0
<5>[ 1956.055899] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 1956.058593] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
<7>[ 2465.910711] [drm:intel_atomic_check [i915]] New voltage level calculated to be logical 13, actual 13
<7>[ 2465.910784] [drm:bxt_get_dpll [i915]] [CRTC:58:pipe B] using pre-allocated PORT PLL A
<7>[ 2465.910848] [drm:intel_reference_shared_dpll [i915]] using PORT PLL A for pipe B
Followed by "stray"
run.log:
running: igt/kms_flip/vblank-vs-modeset-suspend
[15/72] skip: 4, pass: 10, fail: 1 \
FATAL: command execution failed
java.io.EOFException
...
Finished: FAILURE
Completed CI_IGT_test CI_DRM_3364/shard-glkb2/26 : FAILURE
CI_IGT_test runtime 687 seconds
Rebooting shard-glkb2@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/IGT_3997/shard-glkb6/igt@kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-pwrite.html
dmesg:
<5>[ 1874.323543] owatch: Using watchdog device /dev/watchdog0
<5>[ 1874.323619] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 1874.324665] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
<7>[ 2378.064206] [drm:intel_atomic_commit_tail [i915]] [CRTC:42:pipe A]
<7>[ 2378.064339] [drm:verify_single_dpll_state.isra.75 [i915]] PORT PLL A
<7>[ 2378.250481] [IGT] kms_frontbuffer_tracking: executing
run.log:
running: igt/drv_suspend/forcewake-hibernate
[12/73] skip: 1, pass: 9, fail: 2 |
FATAL: command execution failed
java.io.EOFException
...
Finished: FAILURE
Completed CI_IGT_test CI_DRM_3368/shard-glkb6/6 : FAILURE
CI_IGT_test runtime 500 seconds
Rebooting shard-glkb6
run.log is weird it thinks it is still running igt/drv_suspend/forcewake-hibernate, allthough dmesg and results notice we have started kms_frontbuffer_tracking.@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3374/shard-glkb6/igt@kms_flip@flip-vs-panning-vs-hang.html
dmesg:
<5>[ 2808.875155] owatch: Using watchdog device /dev/watchdog0
<5>[ 2808.875235] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 2808.876269] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
<7>[ 3076.548712] [drm:gen8_irq_handler [i915]] hotplug event received, stat 0x00000008, dig 0x1a001818, pins 0x00000010
<7>[ 3076.548831] [drm:intel_hpd_irq_handler [i915]] digital hpd port A - long
<7>[ 3076.548935] [drm:intel_hpd_irq_handler [i915]] Re
"<7>[ 3076.548935] [drm:intel_hpd_irq_handler [i915]] Re" has clearly been cut-off bug stopped execution.
run.log:
running: igt/kms_flip/vblank-vs-modeset-suspend
[06/75] skip: 1, pass: 4, fail: 1 -
FATAL: command execution failed
java.io.EOFException
...
Finished: FAILURE
Completed CI_IGT_test CI_DRM_3374/shard-glkb6/32 : FAILURE
CI_IGT_test runtime 378 seconds
Rebooting shard-glkb6@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4010/shard-glkb5/igt@kms_fbcon_fbt@fbc-suspend.html
dmesg:
<5>[20391.277009] owatch: Using watchdog device /dev/watchdog0
<5>[20391.277089] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[20391.278213] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
<7>[20413.415648] [drm:verify_connector_state.isra.75 [i915]] [CONNECTOR:77:eDP-1]
<7>[20413.415736] [drm:intel_atomic_commit_tail [i915]] [CRTC:43:pipe A]
<7>[20413.415985] [drm:verify_single_dpll_state.isra.76 [i915]] PORT PLL A
run.log:
running: igt/kms_fbcon_fbt/fbc-suspend
[09/75] skip: 1, pass: 8 /
FATAL: command execution failed
java.io.EOFException
...
Finished: FAILURE
Completed CI_IGT_test CI_DRM_3378/shard-glkb5/20 : FAILURE
CI_IGT_test runtime 278 seconds
Rebooting shard-glkb5@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3383/fi-glk-dsi/igt@gem_exec_gttfill@basic.html
This is a straing-up glk-dsi incomplete no GPU hung before:
dmesg:
<5>[ 24.027629] owatch: Using watchdog device /dev/watchdog0
<5>[ 24.028503] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 24.031168] owatch: timeout for /dev/watchdog0 set to 100 (requested 100)
...
<7>[ 205.013427] [IGT] gem_exec_flush: starting subtest basic-wb-rw-default
<7>[ 210.403714] [IGT] gem_exec_flush: exiting, ret=0
<7>[ 210.693921] [IGT] gem_exec_flush: executing
Followed by "stray"
run.log:
running: igt/gem_exec_gttfill/basic
[065/289] skip: 7, pass: 58 /
FATAL: command execution failed
java.io.EOFException
...
Finished: FAILURE
Completed CI_IGT_test CI_DRM_3383/fi-glk-dsi/0 : FAILURE
CI_IGT_test runtime 252 seconds
Rebooting fi-glk-dsi@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3389/fi-glk-dsi/igt@gem_exec_reloc@basic-write-read-active.html
dmesg:
<5>[ 31.994582] owatch: Using watchdog device /dev/watchdog0
<5>[ 31.994798] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 31.997674] owatch: timeout for /dev/watchdog0 set to 100 (requested 100)
...
<7>[ 252.326077] [IGT] gem_exec_reloc: starting subtest basic-write-cpu-active
<4>[ 252.439204] Setting dangerous option reset - tainting kernel
<7>[ 252.439892] [IGT] gem_exec_reloc: exiting, ret=0
Followed by "stray"
run.log:
running: igt/gem_exec_reloc/basic-write-read-active
[097/289] skip: 10, pass: 87 /
FATAL: command execution failed
java.io.EOFException
...
Finished: FAILURE
Completed CI_IGT_test CI_DRM_3389/fi-glk-dsi/0 : FAILURE
CI_IGT_test runtime 314 seconds
Rebooting fi-glk-dsi@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3379/shard-glkb3/igt@kms_flip@render-flip-vs-panning-interruptible.html
dmesg:
<5>[ 37.400972] owatch: Using watchdog device /dev/watchdog0
<5>[ 37.401191] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 37.402407] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
<7>[ 437.056593] [drm:drm_mode_setcrtc] [CONNECTOR:77:eDP-1]
<7>[ 437.058583] [drm:drm_mode_setcrtc] [CRTC:43:pipe A]
<7>[ 437.058654] [drm:drm_mode_setcrtc] [CONNECTOR:77:eDP-1]
Followed by "stray".
run.log:
running: igt/gem_softpin/noreloc-s3
[42/75] skip: 16, pass: 23, fail: 3 -
FATAL: command execution failed
java.io.EOFException
...
Finished: FAILURE
Completed CI_IGT_test CI_DRM_3379/shard-glkb3/1 : FAILURE
CI_IGT_test runtime 548 seconds
Rebooting shard-glkb3@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4011/shard-glkb6/igt@kms_render@direct-render.html
dmesg:
<5>[ 41.747115] owatch: Using watchdog device /dev/watchdog0
<5>[ 41.747344] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 41.748600] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
<7>[ 480.693681] [drm:intel_panel_enable_backlight [i915]] pipe C
<7>[ 480.693736] [drm:intel_panel_actually_set_backlight [i915]] set backlight PWM = 67764
<7>[ 480.693799] [drm:intel_enable_ddi [i915]] Panel doesn't support DRRS
run.log:
running: igt/kms_flip/vblank-vs-dpms-suspend-interruptible
[17/75] skip: 6, pass: 9, fail: 2 /
FATAL: command execution failed
java.io.EOFException
...
Finished: FAILURE
Completed CI_IGT_test CI_DRM_3380/shard-glkb6/10 : FAILURE
CI_IGT_test runtime 558 seconds
Rebooting shard-glkb6@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3385/shard-glkb6/igt@gem_exec_create@forked.html
dmesg:
<5>[ 992.009972] owatch: Using watchdog device /dev/watchdog0
<5>[ 992.010074] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 992.011400] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
<7>[ 1589.117158] [drm:intel_power_well_disable [i915]] disabling DC off
<7>[ 1589.117198] [drm:gen9_enable_dc5 [i915]] Enabling DC5
<7>[ 1589.117264] [drm:gen9_set_dc_state [i915]] Setting DC state from 00 to 01
run.log:
running: igt/drv_suspend/fence-restore-tiled2untiled-hibernate
[36/74] skip: 11, pass: 21, fail: 4 |
FATAL: command execution failed
java.io.EOFException
...
Finished: FAILURE
Completed CI_IGT_test CI_DRM_3385/shard-glkb6/34 : FAILURE
CI_IGT_test runtime 704 seconds
Rebooting shard-glkb6@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3419/shard-glkb6/igt@gem_partial_pwrite_pread@write-uncached.html
another system hang looking incomplete@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3442/shard-glkb6/igt@kms_atomic_transition@plane-all-transition.html
dmesg:
<5>[ 419.687780] owatch: Using watchdog device /dev/watchdog0
<5>[ 419.687938] owatch: Watchdog /dev/watchdog0 is a software watchdog
<5>[ 419.689051] owatch: timeout for /dev/watchdog0 set to 370 (requested 370)
...
<7>[ 815.647724] [drm:gen9_set_dc_state [i915]] Setting DC state from 00 to 01
<7>[ 816.771154] [IGT] gem_exec_blt: exiting, ret=0
<7>[ 817.163235] [IGT] kms_atomic_transition: executing
Followed by "stray"
run.log:
running: igt/gem_workarounds/hibernate-resume-fd
[02/75] pass: 2 -
FATAL: command execution failed
java.io.EOFException
...
Finished: FAILURE
Completed CI_IGT_test CI_DRM_3442/shard-glkb6/8 : FAILURE
CI_IGT_test runtime 432 seconds
Rebooting shard-glkb6
something weird is going on since the test reported doesn't match running test in run.log.@martasaid:nothing obviously suspicious in dmesg, no timeout indication in run.log
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3451/shard-glkb6/igt@kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-fullscreen.html
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3451/shard-glkb5/igt@gem_eio@in-flight-suspend.html
https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4032/shard-glkb6/igt@gem_pwrite_pread@display-pwrite-blt-gtt_mmap-performance.html
https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4034/shard-glkb6/igt@pm_rpm@drm-resources-equal.html
https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4036/shard-glkb6/igt@kms_cursor_legacy@pipe-a-forked-move.html@martasaid:https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4033/shard-glkb1/igt@kms_flip@flip-vs-modeset-vs-hang-interruptible.html
weird looking run.log:
[66/76] skip: 21, pass: 37, fail: 8 -
Build timed out (after 17 minutes). Marking the build as aborted.
pass: igt/kms_cursor_crc/cursor-256x256-suspend
[67/76] skip: 21, pass: 38, fail: 8 -
running: igt/gem_partial_pwrite_pread/writes-after-reads
[67/76] skip: 21, pass: 38, fail: 8 \
Set build name.
New build name is 'CI_DRM_3448/shard-glkb1/4'
Variable with name 'BUILD_DISPLAY_NAME' already exists, current value: 'CI_DRM_3448/shard-glkb1/4', new value: 'CI_DRM_3448/shard-glkb1/4'
SSH: Connecting from host [glkb1]
SSH: Connecting with configuration [archive] ...
pass: igt/gem_partial_pwrite_pread/writes-after-reads
[68/76] skip: 21, pass: 39, fail: 8 <br> running: igt/kms_flip/flip-vs-modeset-vs-hang-interruptible
[68/76] skip: 21, pass: 39, fail: 8 |
SSH: Disconnecting configuration [archive] ...
SSH: Transferred 75 file(s)
Build was aborted
Notifying upstream projects of job completion
Finished: ABORTED
Completed CI_IGT_test CI_DRM_3448/shard-glkb1/4 : ABORTED
CI_IGT_test runtime 1084 seconds
Rebooting shard-glkb1
from dmesg:
<6>[18726.272872] r8169 0000:01:00.0 enp1s0: link down
