ci/build_functions: Improve detection of whether a container does not exist or a network error happened
+ skopeo inspect docker://registry.freedesktop.org/mupuf/valve-infra/valve-infra-base-container:2023-01-16.1
+ jq '[.Digest, .Layers]'
time="2023-01-31T12:06:16Z" level=fatal msg="Get \"https://storage.googleapis.com/fdo-gitlab-registry/docker/registry/v2/blobs/sha256/39/3995645b1ad17ae3bca82fea6f4ec5297b80f9fd017ef07ddd9f311157ff12fa/data?Expires=1675167966&GoogleAccessId=fdo-gitlab-registry-upload%40fdo-gitlab.iam.gserviceaccount.com&Signature=kCnLQJ5J9luQyhn5V9xTA1VcodyBw%2BhXAacqM13aLIofQkuflCOl%2FJkxM9iKinhVZHOEx21GGcv9ok%2BrEl4jDy%2FkV9dWT6i2bjZ9n3q%2F%2F9Pl2JkWDuR7e56rc7NbJ51ofdDQYnUJiV%2B3ExmskU3PgV2eatZ5aVM%2BhD0G4K6oxr8D120BY0XIrG5u9tu5Ppbz4HTb1HVx3o54b5I80dtaSPDVuPDDvWT0ddhp0hKnM2UezOIglwZVVX0yu5YDYSyWCm95R2gAqCr0RNWICKufOhImTG%2Fgg%2BjHJVu4MnVp%2FoX7jV0i0hunt7MpxLfeR6wcZGK4m0g%2BX9hlcO2LKTBAqA%3D%3D\": net/http: TLS handshake timeout"
+ true
Source: https://gitlab.freedesktop.org/mupuf/valve-infra/-/jobs/35610086#L37
Here is a local reproduction of the issue:
# Missing container image
$ skopeo inspect docker://registry.freedesktop.org/mupuf/valve-infra/valve-infra-base-container:2023-01-16.2; echo $?
FATA[0001] Error parsing image name "docker://registry.freedesktop.org/mupuf/valve-infra/valve-infra-base-container:2023-01-16.2": reading manifest 2023-01-16.2 in registry.freedesktop.org/mupuf/valve-infra/valve-infra-base-container: manifest unknown
1
# Missing connectivity
$ skopeo inspect docker://registry.freedesktop.org/mupuf/valve-infra/valve-infra-base-container:2023-01-16.1; echo $?
FATA[0000] Error parsing image name "docker://registry.freedesktop.org/mupuf/valve-infra/valve-infra-base-container:2023-01-16.1": pinging container registry registry.freedesktop.org: Get "https://registry.freedesktop.org/v2/": dial tcp: lookup registry.freedesktop.org: Temporary failure in name resolution
1
Since we can't use the exit code to figure out if the error comes from network connectivity or a missing container, we'll have to try parsing and look for "manifest unknown" in the output.
If the substring is missing and exit code != 0, we should retry a couple more times with a growing sleep time before simply failing the job.
In no case should we start creating a new image unless we know for sure that the image is missing.
Edited by Martin Roukala