Fix intermediate XOpenDisplay CI failures

The CI intermediately fails with "WAFFLE_ERROR_UNKNOWN: XOpenDisplay failed", despite the 10 second sleep in the xvfb-run invocation. Never seen this on my machines, so I suspect there's something CI specific in there.

Investigate or at least reduce the likelihood of it happening.