I have seen job folder compression on the CI gateway take 30+ seconds, and using 100% of many cores when multiple jobs start at the same time.
This is a little silly since, not only do we waste time compressing the folder, but we then need to decompress it right after!
The reason why compression was enabled by default previously was to support interactive sessions from a developer's workstation. The objective was to reduce the upload time of the share folder by doing light levels of compression (gzip)... which turned out to be waaaay heavier than expected for multiple reasons:
- Python's tarfile defaults to level 9, unlike tar which defaults to 6
- Gzip's performance is abysmal, even at level 1!
Indeed, on my machine, compressing a mostly-incompressible folder
weighing ~250 MB takes 4.7s using gzip -9 and 4.1s for gzip -1.
This should be compared with the compression time of 0.1s when skipping
compression altogether!
Since both CI jobs and interactive developer sessions often share already-compressed tarballs, and the fact that I am pretty much the only one using interactive sessions (on a local network), compression seem to be hurting us more than it is helping.
I would however like to revisit this claim when python add supports for LZ4, since its brings a 10x improvement in compression speed while providing compression for the cases where it may have helped, and without increasing the size otherwise.
Fixes: #171 (closed)