ci: Reuse xserver created by meson-dist job in driver build jobs

Currently each driver build job also builds the server. This is bad use
of resources because server build takes way more resources compared to a
driver build. Reusing already built xserver would resolve the issue.