[CI][Runner] Abort on network down

Submitted by Martin Peres

Assigned to Petri Latvala

Link to original bug (#108767)

Description

When the network is down, the external watchdogs are bound to come and kill the machine.

Protect ourselves against these random incompletes by stopping the execution early and exiting.

The current plan was to allow the configuration of a host to ping before executing a new binary to make sure that the network is still up. The host to ping should be specified in igtrc. If none are specified, no pings should be made.

added priority::high severity::normal + 1 deleted label

Martin Peres said:

https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_5146/fi-skl-6700k2/igt@i915_selftest@live_workarounds.html

https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4716/fi-skl-6700k2/igt@drv_selftest@live_contexts.html

Both runs got killed because of the network issue

Jani Saarinen said:

Dropping priority, new feature request.

Martin Peres said:

https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_5244/fi-kbl-7560u/igt@gem_basic@create-close.html

No logs, but it seems like it got stuck very early on, as can be seen in the run log:

[001/332] (960s left) core_auth (basic-auth)
FATAL: command execution failed
java.io.EOFException
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2681)
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3156)
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:862)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
at hudson.remoting.Command.readFrom(Command.java:140)
at hudson.remoting.Command.readFrom(Command.java:126)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Caused: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
Caused: java.io.IOException: Backing channel 'fi-kbl-7560u' is disconnected.
at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:214)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
at com.sun.proxy.$Proxy64.isAlive(Unknown Source)
at hudson.Launcher

$RemoteLauncher$

ProcImpl.isAlive(Launcher.java:1144)
at hudson.Launcher

RemoteLauncher

ProcImpl.join(Launcher.java:1136)
at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:155)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
at hudson.model.Build$BuildExecution.build(Build.java:206)
at hudson.model.Build$BuildExecution.doRun(Build.java:163)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
at hudson.model.Run.execute(Run.java:1810)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
FATAL: Unable to delete script file /tmp/jenkins3295738120970564664.sh
java.io.EOFException
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2681)
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3156)
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:862)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
at hudson.remoting.Command.readFrom(Command.java:140)
at hudson.remoting.Command.readFrom(Command.java:126)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Caused: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
Caused: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on fi-kbl-7560u failed. The channel is closing down or has closed down
at hudson.remoting.Channel.call(Channel.java:948)
at hudson.FilePath.act(FilePath.java:1070)
at hudson.FilePath.act(FilePath.java:1059)
at hudson.FilePath.delete(FilePath.java:1563)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:123)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
at hudson.model.Build$BuildExecution.build(Build.java:206)
at hudson.model.Build$BuildExecution.doRun(Build.java:163)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
at hudson.model.Run.execute(Run.java:1810)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Build step 'Execute shell' marked build as failure
Set build name.
Workspace is not accessible
ERROR: no workspace for CI_IGT_test #2662333
Notifying upstream projects of job completion
Finished: FAILURE
Completed CI_IGT_test CI_DRM_5244/fi-kbl-7560u/0 : FAILURE
CI_IGT_test runtime 6 seconds
Rebooting fi-kbl-7560u

Lakshmi closed a related bug:

*** Bug 108953 has been marked as a duplicate of this bug. ***

Lakshmi closed a related bug:

*** Bug 108680 has been marked as a duplicate of this bug. ***

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-kbl-7560u: random tests - incomplete
fi-kbl-7560u: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_172/fi-kbl-7560u/igt@pm_lpsp@edp-panel-fitter.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-kbl-7560u: random tests - incomplete
fi-kbl-7560u: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_175/fi-kbl-7560u/igt@kms_cursor_crc@cursor-128x128-offscreen.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-kbl-7560u: random tests - incomplete
fi-kbl-7560u: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_5403/fi-kbl-7560u/igt@gem_close_race@basic-threads.html

CI Bug Log said:

The CI Bug Log issue associated to this bug has been updated.

New filters associated

* SKL: random tests - incomplete
- https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_5464/shard-skl8/igt@kms_flip@flip-vs-panning-vs-hang-interruptible.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-kbl-7560u: random tests - incomplete
fi-kbl-7560u: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4783/fi-kbl-7560u/igt@kms_pipe_crc_basic@suspend-read-crc-pipe-b.html
  
• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_194/fi-kbl-7560u/igt@kms_pipe_crc_basic@suspend-read-crc-pipe-b.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-kbl-7560u: random tests - incomplete
fi-kbl-7560u: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_177/fi-kbl-7560u/igt@prime_busy@wait-hang-render.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-kbl-7560u: random tests - incomplete
fi-kbl-7560u: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_180/fi-kbl-7560u/igt@kms_pipe_b_c_ivb@enable-pipe-c-while-b-has-3-lanes.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

CHAMELIUM: igt@kms_chamelium@* - fail - Last errno: 113, No route to host
CHAMELIUM: igt@kms_chamelium@* - fail - Last errno: 113, No route to host

No new failures caught with the new filter

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-cfl-8109u: random tests - incomplete
fi-cfl-8109u: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_178/fi-cfl-8109u/igt@kms_flip@2x-flip-vs-blocking-wf-vblank.html

CI Bug Log said:

The CI Bug Log issue associated to this bug has been updated.

New filters associated

* fi-blb-e6850: random tests - incomplete
- https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_178/fi-blb-e6850/igt@kms_atomic_interruptible@universal-setplane-cursor.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-kbl-7560u: random tests - incomplete
fi-kbl-7560u: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_198/fi-kbl-7560u/igt@kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-move.html
  
• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_200/fi-kbl-7560u/igt@kms_frontbuffer_tracking@fbc-rgb101010-draw-mmap-gtt.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-cfl-8109u: random tests - incomplete
fi-cfl-8109u: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_187/fi-cfl-8109u/igt@kms_flip@2x-dpms-vs-vblank-race-interruptible.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-cfl-8109u: random tests - incomplete
fi-cfl-8109u: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_192/fi-cfl-8109u/igt@kms_busy@basic-modeset-c.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-cfl-8109u: random tests - incomplete
fi-cfl-8109u: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_195/fi-cfl-8109u/igt@perf_pmu@enable-race-vcs1.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-blb-e6850: random tests - incomplete
fi-blb-e6850: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_202/fi-blb-e6850/igt@kms_flip@flip-vs-suspend.html

CI Bug Log said:

The CI Bug Log issue associated to this bug has been updated.

New filters associated

* fi-snb-2520m: igt@gem_eio@throttle - incomplete - empty logs
- https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_202/fi-snb-2520m/igt@gem_eio@throttle.html

CI Bug Log said:

The CI Bug Log issue associated to this bug has been updated.

New filters associated

* fi-kbl-guc: igt@kms_frontbuffer_tracking@fbcpsr-2p-indfb-fliptrack - incomplete - empty logs
- https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_201/fi-kbl-guc/igt@kms_frontbuffer_tracking@fbcpsr-2p-indfb-fliptrack.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-kbl-7560u: random tests - incomplete
fi-kbl-7560u: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_203/fi-kbl-7560u/igt@gem_pwrite_pread@display-pwrite-blt-gtt_mmap-performance.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-snb-2520m: igt@gem_eio@throttle - incomplete - empty logs
fi-snb-2520m: igt@* - incomplete - empty logs

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_203/fi-snb-2520m/igt@kms_color@pipe-a-legacy-gamma.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-snb-2520m: igt@* - incomplete - empty logs
fi-snb-2520m: igt@* - incomplete - empty logs

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_211/fi-snb-2520m/igt@kms_busy@extended-pageflip-modeset-hang-oldfb-render-a.html

CI Bug Log said:

The CI Bug Log issue associated to this bug has been updated.

New filters associated

* fi-kbl-8809g: igt@* - incomplete - empty logs
- https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_211/fi-kbl-8809g/igt@kms_busy@extended-pageflip-modeset-hang-oldfb-render-a.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-kbl-guc: igt@kms_frontbuffer_tracking@fbcpsr-2p-indfb-fliptrack - incomplete - empty logs
fi-kbl-guc: i random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_213/fi-kbl-guc/igt@gem_exec_basic@basic-vebox.html
  
• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_215/fi-kbl-guc/igt@gem_exec_basic@basic-vebox.html

CI Bug Log said:

The CI Bug Log issue associated to this bug has been updated.

New filters associated

* fi-skl-6770: random tests - incomplete
- https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_215/fi-skl-6770hq/igt@gem_exec_basic@basic-vebox.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-kbl-guc: i random tests - incomplete
fi-kbl-guc: random tests - incomplete

No new failures caught with the new filter

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-kbl-guc: random tests - incomplete
fi-kbl-guc: random tests - incomplete

No new failures caught with the new filter

CI Bug Log said:

The CI Bug Log issue associated to this bug has been updated.

New filters associated

* fi-kbl-guc: random tests - incomplete - empty logs
- https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_209/fi-kbl-guc/igt@kms_frontbuffer_tracking@fbc-rgb565-draw-pwrite.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-cfl-8109u: random tests - incomplete
fi-cfl-8109u: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_210/fi-cfl-8109u/igt@kms_busy@extended-modeset-hang-newfb-with-reset-render-b.html

CI Bug Log said:

The CI Bug Log issue associated to this bug has been updated.

New filters associated

* fi-ilk-650: random tests - incomplete
- https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_217/fi-ilk-650/igt@gem_exec_await@wide-all.html

CI Bug Log said:

The CI Bug Log issue associated to this bug has been updated.

New filters associated

* BDW: random tests - incomplete
- https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_218/fi-bdw-5557u/igt@perf_pmu@enable-race-vecs0.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-cfl-8109u: random tests - incomplete
fi-cfl-8109u: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_218/fi-cfl-8109u/igt@kms_busy@extended-modeset-hang-oldfb-with-reset-render-b.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-kbl-guc: random tests - incomplete - empty logs
fi-kbl-guc: random tests - incomplete - empty logs

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_218/fi-kbl-guc/igt@kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-shrfb-draw-mmap-gtt.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

CHAMELIUM: igt@kms_chamelium@* - fail - Last errno: 113, No route to host
CHAMELIUM: igt@kms_chamelium@* - fail - Last errno: 113, No route to host

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_222/fi-kbl-7500u/igt@kms_chamelium@hdmi-cmp-yu16.html
  
• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_223/fi-kbl-7500u/igt@kms_chamelium@hdmi-cmp-yu16.html
  
• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_223/fi-kbl-7567u/igt@kms_chamelium@hdmi-cmp-yu16.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-snb-2520m: igt@* - incomplete - empty logs
fi-snb-2520m: igt@* - random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_221/fi-snb-2520m/igt@kms_plane@plane-position-covered-pipe-a-planes.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-cfl-8109u: random tests - incomplete
fi-cfl-8109u: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_224/fi-cfl-8109u/igt@kms_rotation_crc@multiplane-rotation.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-kbl-7560u: random tests - incomplete
fi-kbl-7560u: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_225/fi-kbl-7560u/igt@gem_exec_parallel@default.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-cfl-8109u: random tests - incomplete
fi-cfl-8109u: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_227/fi-cfl-8109u/igt@kms_flip@flip-vs-rmfb.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-cfl-8109u: random tests - incomplete
fi-cfl-8109u: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_228/fi-cfl-8109u/igt@kms_rotation_crc@multiplane-rotation-cropping-top.html

CI Bug Log said:

The CI Bug Log issue associated to this bug has been updated.

New filters associated

* HSW: Random tests - incomplete
- https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_227/fi-hsw-4770/igt@perf_pmu@busy-double-start-vcs0.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-snb-2520m: igt@* - random tests - incomplete
fi-snb-2520m: igt@* - random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_232/fi-snb-2520m/igt@kms_flip@basic-plain-flip.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

fi-kbl-7560u: random tests - incomplete
fi-kbl-7560u: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_234/fi-kbl-7560u/igt@kms_flip@plain-flip-interruptible.html

CI Bug Log said:

The CI Bug Log issue associated to this bug has been updated.

New filters associated

* fi-icl-y : random tests - incomplete
- https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_5698/fi-icl-y/igt@debugfs_test@read_all_entries.html

CI Bug Log said:

A CI Bug Log filter associated to this bug has been updated:

SKL: random tests - incomplete
SKL: random tests - incomplete

New failures caught by the filter:

• https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_5717/shard-skl8/igt@kms_vblank@pipe-c-query-forked-hang.html

CI Bug Log said:

The CI Bug Log issue associated to this bug has been updated.

New filters associated

* fi-icl-u2: Random tests - incomplete
- https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_5776/fi-icl-u2/igt@gem_basic@create-close.html

CI Bug Log said:

The CI Bug Log issue associated to this bug has been updated.

New filters associated

* shard-iclb1: Random tests - in complete- empty logs
- https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_5784/shard-iclb1/igt@kms_plane_scaling@pipe-c-scaler-with-pixel-format.html