ci: start unifying LAVA and bare-metal
There is a whole heap of stuff in here, but ultimately the goal is to make LAVA more like bare-metal.
When LAVA was built up, it was focused on the least impact to the already-fragile system around it, as well as keeping the hardware/tests reliable (as we were the first to be running on real hardware) rather than minimising the impedance mismatch. Since then, bare-metal has arrived which has a much smaller impedance mismatch with GitLab CI, but on the other hand bare-metal and LAVA are completely divergent. This sucks.
bare-metal ultimately has the right answer of being less out-of-band than LAVA currently is. This series has a bunch of changes to LAVA to fit in with what bare-metal has built up, and shares more code in the process.
The main visible result is that we don't get griefed by YAML->bash->Jinja->YAML->Python->YAML->bash variable quoting, which seems like it couldn't possibly go wrong, but somehow has. As a nice side effect, we don't have to duplicate the list of hardware-relevant environment variables.
There's a lot more to be done with sharing the init script instead of having the same thing be formatted once in bash and once in YAML, but I hit a wall after I had dinner and this series is already large enough. It's a pretty straightforward and obvious next step for tomorrow though. There's also a lot more to be done to unpick some of the weirdness around the container builds and variables in particular, but that's hairy enough that it's tomorrow++.
Based on (and includes reviewed commits from) !11293 (merged).