Executor: handle EasySNMPError better
When I am in the configuration interface of my PDU, SNMP keeps failing. We can't do anything about that (aside from adding all the functionality we need in the dashboard), so let's try to handle this error better, and even add retry support?
Oct 17 07:18:57 mupuf-gateway executor[23734]: Traceback (most recent call last):
Oct 17 07:18:57 mupuf-gateway executor[23734]: File "/usr/lib/python3.10/site-packages/valve_gfx_ci/executor/server/executor.py", line 679, in run
Oct 17 07:18:57 mupuf-gateway executor[23734]: session_init()
Oct 17 07:18:57 mupuf-gateway executor[23734]: File "/usr/lib/python3.10/site-packages/valve_gfx_ci/executor/server/executor.py", line 533, in session_init
Oct 17 07:18:57 mupuf-gateway executor[23734]: self.pdu_port.set(PDUState.OFF)
Oct 17 07:18:57 mupuf-gateway executor[23734]: File "/usr/lib/python3.10/site-packages/valve_gfx_ci/executor/server/pdu/__init__.py", line 60, in set
Oct 17 07:18:57 mupuf-gateway executor[23734]: self.pdu.set_port_state(self.port_id, state)
Oct 17 07:18:57 mupuf-gateway executor[23734]: File "/usr/lib/python3.10/site-packages/valve_gfx_ci/executor/server/pdu/drivers/snmp.py", line 29, in retry
Oct 17 07:18:57 mupuf-gateway executor[23734]: return func(*args, **kwargs)
Oct 17 07:18:57 mupuf-gateway executor[23734]: File "/usr/lib/python3.10/site-packages/valve_gfx_ci/executor/server/pdu/drivers/snmp.py", line 125, in set_port_state
Oct 17 07:18:57 mupuf-gateway executor[23734]: ret = snmp_set(self.outlet_ctrl_oid(port_id),
Oct 17 07:18:57 mupuf-gateway executor[23734]: File "/usr/lib/python3.10/site-packages/easysnmp/easy.py", line 43, in snmp_set
Oct 17 07:18:57 mupuf-gateway executor[23734]: return session.set(oid, value, type)
Oct 17 07:18:57 mupuf-gateway executor[23734]: File "/usr/lib/python3.10/site-packages/easysnmp/session.py", line 348, in set
Oct 17 07:18:57 mupuf-gateway executor[23734]: success = interface.set(self, varlist)
Oct 17 07:18:57 mupuf-gateway executor[23734]: easysnmp.exceptions.EasySNMPError: (genError) A general failure occured
Oct 17 07:18:59 mupuf-gateway executor[23734]: Traceback (most recent call last):
Oct 17 07:18:59 mupuf-gateway executor[23734]: File "/usr/bin/executor", line 8, in <module>
Oct 17 07:18:59 mupuf-gateway executor[23734]: sys.exit(run())
Oct 17 07:18:59 mupuf-gateway executor[23734]: File "/usr/lib/python3.10/site-packages/valve_gfx_ci/executor/server/__init__.py", line 29, in run
Oct 17 07:18:59 mupuf-gateway executor[23734]: run(config_f=args.config, socket_path=args.socket)
Oct 17 07:18:59 mupuf-gateway executor[23734]: File "/usr/lib/python3.10/site-packages/valve_gfx_ci/executor/server/executor.py", line 811, in run
Oct 17 07:18:59 mupuf-gateway executor[23734]: executor.run()
Oct 17 07:18:59 mupuf-gateway executor[23734]: File "/usr/lib/python3.10/site-packages/valve_gfx_ci/executor/server/executor.py", line 685, in run
Oct 17 07:18:59 mupuf-gateway executor[23734]: session_end()
Oct 17 07:18:59 mupuf-gateway executor[23734]: File "/usr/lib/python3.10/site-packages/valve_gfx_ci/executor/server/executor.py", line 543, in session_end
Oct 17 07:18:59 mupuf-gateway executor[23734]: self.job_console.close()
Oct 17 07:18:59 mupuf-gateway executor[23734]: AttributeError: 'NoneType' object has no attribute 'close'