Speeding up the cerbero MSVC non-deps build
Looking at https://gitlab.freedesktop.org/nirbheek/cerbero/-/jobs/56701480 which was a job that happened at a low CI utilization time, some things jumped out at me:
- 45s to
cp -a
cerbero-sources cache - ??s to
du -sch cerbero-sources
(1st time) - 2 min to run
cargo vendor
for cargo-c (cargo-c -> fetch
) out of ~3 min to runfetch-bootstrap
(incl 45s to download rust toolchain) - 3 min to run
cargo vendor
for gst-plugins-rs (gst-plugins-rs -> fetch
) out of ~5 min to runfetch-package
- 30s to run
du -sch cerbero-sources
(2nd time) - 1.5 min to unpack the deps cache (3s to download it)
- 45s to install the latest rust toolchain in
bootstrap
- 1 min to re-install meson in
bootstrap
(!?) - 6.5 min to build and install the gstreamer monorepo
- 10 min to build and install gst-plugins-rs (using cargo-c)
- 1 min to package the results to tar.xz
- Total: 32 min
Total job time: 32 min
The time needed to download and initialize the container is not included in this, but it is definitely quite slow.
Mitigations:
-
cargo vendor
actually runs twice- The first time, it is run during
fetch
after extracting the git repo/tarball so we can fetch everything from the internet - The second time, it is run during
extract
to ensure that everything is available in the source tree for an offline build - This happens because fetch and extract are two separate steps. One fix is to add a mechanism that tells us to skip the extract phase if fetch completed and the recipe hasn't changed since fetch completed.
- The first time, it is run during
-
cargo vendor
andgst-plugins-rs
builds taking so long are probably related: the windows VM runs on top of the current stable version of qemu-kvm with virtio, which is probably quite slow at I/O.- QEMU 9.0 might fix this, but it hasn't been released yet: https://blog.vmsplice.net/2024/01/qemu-aiocontext-removal-and-how-it-was.html
- Another option is to passthrough a device into the VM so it bypasses virtio and run everything inside that
- This will also help with provisioning of the container, which is I/O bottlenecked
- Disabling Rust debug symbols and turning off optimizations might speed up gst-plugins-rs
- If the issue is I/O, doing a full-debug build without optimizations might slow things down because it'll increase the size of artifacts
- Investigate sccache for Rust on Windows
- This will be I/O-heavy to copy into/out of the container, so it will have to be mounted from the runner itself