Public
Authored by Arkadiusz Hiler

Why I do not use wayland/ci-templates?

Manual Tagging vs Automatic Rebuilding

With wayland/ci-templates it's not enough to just change the script that is used to build an image, you also have to bump the tag:

DEBIAN_TAG: 2019-03-29-01
DEBIAN_EXEC: 'bash .gitlab-ci/debian-install.sh'

Instead I prefer to have the new image built automatically whenever relevant Dockerfile is touched:

DOCKERFILE=$1
IMAGENAME=$2

DOCKERFILE_CHECKSUM=$(sha1sum $DOCKERFILE | cut -d ' ' -f1)
REGISTRYPATH=$CI_REGISTRY/$CI_PROJECT_PATH/$IMAGENAME:dockerfile-$DOCKERFILE_CHECKSUM

# base container (building, etc) - we rebuild only if changed or forced
skopeo inspect docker://$REGISTRYPATH
IMAGE_PRESENT=$?

if [ $IMAGE_PRESENT -eq 0 ]; then
	echo "Skipping, already built"
else
	echo "Building!"
	podman build --build-arg=CI_COMMIT_SHA=$CI_COMMIT_SHA \
		     --build-arg=CI_REGISTRY_IMAGE=$CI_REGISTRY_IMAGE \
		     --squash -t $REGISTRYPATH -f $DOCKERFILE .
	podman push $REGISTRYPATH
fi

so the build step looks more or less like this:

build-containers:build-fedora:
  stage: build-containers
  image: registry.freedesktop.org/wayland/ci-templates/buildah
  script:
    - .gitlab-ci/pull-or-rebuild.sh base Dockerfile.build-fedora build-fedora

Then if I want to rebuild the image I can just bump a magic number in a comment inside of the Dockerfile itself:

# bump this number to force a new image: 1

Everything is neatly contained in a single file.

To know which image to use for the actual execution in one of the following stages of the pipeline we can use the commit SHA as an extra tag on the image:

COMMIT_TAG=commit-$CI_COMMIT_SHA
# $REGISTRYPATH_WITH_COMMIT_TAG=...
skopeo copy docker://$REGSITRYPATH docker://$REGISTRYPATH_WITH_COMMIT_TAG

And then in .gitlab-ci.yml:

image: $CI_REGISTRY/$CI_PROJECT_PATH/build-debian:commit-$CI_COMMIT_SHA

Extra Conventions To Learn

Gitlab CI uses docker images and using Dockerfiles feels familiar if you ever worked on any project on the gitlab.org. wayland/ci-templates introduces another layer of abstraction on top of that:

include:
  - project: 'wayland/ci-templates'
    ref: c73dae8b84697ef18e2dbbf4fed7386d9652b0cd
    file: '/templates/debian.yml'

variables:
  DEBIAN_TAG: 2019-03-29-01
  DEBIAN_VERSION: testing
  TEST_IMAGE: "$CI_REGISTRY_IMAGE/debian/$DEBIAN_VERSION:$DEBIAN_TAG"

container_build:
  extends: .debian@container-ifnot-exists
  stage: containers-build
  variables:
    DEBIAN_EXEC: 'bash .gitlab-ci/debian-install.sh'

This feels pretty opaque and from my experience makes people more hesitant/reluctant to ever touch your .gitlab-ci.yaml. Even when you link to https://gitlab.freedesktop.org/wayland/ci-templates/blob/master/templates/debian.yml#L3 directly (which many project don't do), it feels more magical than having a simpler Dockerfile + a short bash script that handles conditional building available directly in your repo.

What Do I lose?

More?

There is a bit more to my reasoning, as the project discussed here doesn't use merge requests and instead does a pre-merge via mailing list, patchwork and a special repository where changes are pushed under a tag. All this requires quite a bit of weird plumbing which wayland/ci-templates does not fit well, but I'll save you details on those self-inflicted issues as hopefully they will be gone sooner than later.

Once the above is sorted out I would like to have a kind of hybrid of both approaches - being DEBIAN_TAG-less and having the benefit of the curated set of optimizations, possibly through a base image.

Edited
x 1 Byte
  • A hash of the Docker file isn't sufficient to identify an image in general, since the contents of the underlying base image may change over time, so there could end up being multiple different images with the same tag. Maybe it won't matter most of the time, but the more time lies between generating two images using the same Dockerfile, the more likely that there is a relevant difference.

    I'm not sure how bumping the image: line in .gitlab-ci.yml is different from bumping the tag with ci-templates. :) Anyway, the latter was designed to allow generating a new image and making use of it in a single MR.

  • @daenzer:

    A hash of the Docker file isn't sufficient to identify an image in general, since the contents of the underlying base image may change over time, so there could end up being multiple different images with the same tag. Maybe it won't matter most of the time, but the more time lies between generating two images using the same Dockerfile, the more likely that there is a relevant difference.

    Yep. I am aware of this - this happens if you are building the image in two different repositories at different times - packages get updated, base layers change. As long as you are working within a single repository this is not an issue though, as we won't rebuild it if it is there already. To force the rebuild you have to bump the counter inside the Dockerfile effectively changing its checksum.

    If (hopefully) we will move over to MRs this may become an issue and this needs to be changed, but I am not too worried ahead of time.

    @daenzer:

    I'm not sure how bumping the image: line in .gitlab-ci.yml is different from bumping the tag with ci-templates. :) Anyway, the latter was designed to allow generating a new image and making use of it in a single MR.

    You don't bump anything in .gitlab-ci.yml and that is the point - the magic number is inside the Dockerfile itself, so we are still operating on Dockerfile's checksums only.

  • You don't bump anything in .gitlab-ci.yml and that is the point - the magic number is inside the Dockerfile itself, so we are still operating on Dockerfile's checksums only.

    Bumping the magic number generates a new image, but it won't be used until the corresponding image: reference in .gitlab-ci.yml is updated as well. Bumping the tag takes care of both with ci-templates. From where I'm standing, the latter requires less work (changing one line in one file instead of two in two).

  • @daenzer:

    Bumping the magic number generates a new image, but it won't be used until the corresponding image: reference in .gitlab-ci.yml is updated as well. Bumping the tag takes care of both with ci-templates. From where I'm standing, the latter requires less work (changing one line in one file instead of two in two).

    Oh, it will be picked up automatically. You don't have to change anything in the gitlab-ci.yaml. Maybe that was not clear, but this is how it works:

    1. get the SHA1 and check if the image is already in the registry
    • build if it's not there and push
    1. additionaly tag the image with commit-$CI_COMMIT_SHA
    COMMIT_TAG=commit-$CI_COMMIT_SHA
    # $REGISTRYPATH_WITH_COMMIT_TAG=..., some substitution magic
    skopeo copy docker://$REGSITRYPATH docker://$REGISTRYPATH_WITH_COMMIT_TAG
    1. .gitlab-ci.yaml uses this image automatically because we can just use the $CI_COMMIT_SHA in there:
    image: $CI_REGISTRY/$CI_PROJECT_PATH/build-debian:commit-$CI_COMMIT_SHA
  • So basically the checksum here is not treated as a unique identifier no matter where/when you rebuild your image, it's just an automated tag which is not in date format and that you don't have to bump manually.

  • [sneaking in the conversation, and not trying to convince you at all, but just to understand]:

    I understand the idea of the automatic tag (I honestly did not know we could reuse a generated tag), but why would that prevent you from using CI-templates? You could also autogenerate the tag name based on your script, there won't be any differences AFAICT.

    The extra conventions to learn paragraph doesn't really convince me.

    • the include bit is standard gitlab CI, and I really encourage any projects to make use of it, instead of having a single monolithic .gitlab-ci.yml file (like a single main.c vs compilation units)
    • the variables part could be included in the build job, and we could probably add some defaults in ci-templates if this feels too much of a problem. OTOH, this are the most common set of variables the other projects require. Note that the DEBIAN_EXEC variable could be declared there too.

    Last, I find that you summarized it all in the (https://gitlab.freedesktop.org/snippets/700#what-do-i-lose): the whole purpose of ci-templates is to reduce the amount of boilerplate written in the projects .gitlab-ci.yaml, and all of the benefits you listed. I honestly believe the benefits counter balance the drawbacks, and we should probably find a way to accommodate with your particular needs.

    There is one other fundamental advantage for ci-templates: there is a CI for it. It seems silly, but whenever you include an upstream commit from ci-templates, you know that it has been tested for the use you have of it.

    And honestly ci-templates shows its full potential when you start dealing with more than one distribution, especially if this distribution is debian. For debian, you usually need to enable side repos, while with OpenSuSE, Fedora, Arch, you can just declare which pakages you want, include the file, and done, you've got your immutable container...

    [OK, yes, trying to convince you here]

  • @bentiss:

    I understand the idea of the automatic tag (I honestly did not know we could reuse a generated tag), but why would that prevent you from using CI-templates? You could also autogenerate the tag name based on your script, there won't be any differences AFAICT.

    I was considering this as well, sadly I haven't found a way to set the DEBIAN_TAG env variable to sha1sum Dockerfile.build-debian in a way that would satisfy .debian@container-ifnot-exists.

    That would probably require another env variable introduced on the wayland/ci-templates side DEBIAN_TAG_FROM_FILE_SHA or something.

    The extra conventions to learn paragraph doesn't really convince me.

    The main issue here is being a little bit too opaque. It's not a blocker alone, but having this extra layer of indirectness doesn't make things more approachable.

    Last, I find that you summarized it all in the (https://gitlab.freedesktop.org/snippets/700#what-do-i-lose): the whole purpose of ci-templates is to reduce the amount of boilerplate written in the projects .gitlab-ci.yaml, and all of the benefits you listed. I honestly believe the benefits counter balance the drawbacks, and we should probably find a way to accommodate with your particular needs.

    There is one other fundamental advantage for ci-templates: there is a CI for it. It seems silly, but whenever you include an upstream commit from ci-templates, you know that it has been tested for the use you have of it.

    Oh, I am fully aware what I lose - that's why I have acknowledged this in the first place. If you are fine with us adding the following things (or sensible alternatives) to ci-templates, I would be happy to switch:

    • automatic tagging using file checksums
    • generating multiple debian images under different names
    • specifying distro version in the same file that has packages installation steps, so it's all contained in a single file e.g.:
    DEBIAN_VERSION=buster
    
    setup() {
      # add repo
      apt install ..
    }
    

    [OK, yes, trying to convince you here]

    Oh feel free to do this :-)

    If we find a way to do achieve what I have listed above I will be happy to switch and have fewer things to maintain by myself.

  • I was considering this as well, sadly I haven't found a way to set the DEBIAN_TAG env variable to sha1sum Dockerfile.build-debian in a way that would satisfy .debian@container-ifnot-exists.

    That would probably require another env variable introduced on the wayland/ci-templates side DEBIAN_TAG_FROM_FILE_SHA or something.

    Maybe we should just add an eval somewhere before assigning DEBIAN_TAG (though the security implications might be interesting).

    If I understand correctly, DEBIAN_TAG_FROM_FILE_SHA will not solve the issue that (according to the backlog on IRC) gitlab CI won't be able to understand the tag in the image: section of the job...

    But feel free to send a MR that fits your needs.

    Oh, I am fully aware what I lose - that's why I have acknowledged this in the first place. If you are fine with us adding the following things (or sensible alternatives) to ci-templates, I would be happy to switch

    I am maintaining it, but I do not own ci-templates. I basically tried to take the most common denominator in all the projects I was looking at. So if you have different needs, there is no reasons to block a MR. I am not saying it won't be discussed, just that if you have different needs, we should likely be able to accommodate.

    specifying distro version in the same file that has packages installation steps, so it's all contained in a single file e.g.

    I was never fully happy with the way debian is handled in ci-templates. For Debian, I am under the impression that we need to have a script for it, because by default it's not capable of the same level of simplicity than Fedora or Arch for instance. So any improvements in this area would be welcome.

  • Michel Dänzer
    @daenzer started a thread
    Last updated by Arkadiusz Hiler
    Please register or sign in to reply
  • @bentiss:

    If I understand correctly, DEBIAN_TAG_FROM_FILE_SHA will not solve the issue that (according to the backlog on IRC) gitlab CI won't be able to understand the tag in the image: section of the job...

    That is why I have this extra step that tags the image with commit-$CI_COMMIT_SHA, which you can use with image: ..., we would need something like this.

    I am maintaining it, but I do not own ci-templates. I basically tried to take the most common denominator in all the projects I was looking at. So if you have different needs, there is no reasons to block a MR. I am not saying it won't be discussed, just that if you have different needs, we should likely be able to accommodate.

    I'll look at this and I'll try to figure out what I would like to see and hit you with MR or issue.

  • Michel Dänzer
    @daenzer started a thread
    Last updated by Arkadiusz Hiler
    • That is why I have this extra step that tags the image with commit-$CI_COMMIT_SHA, which you can use with image: ..., we would need something like this.

      FWIW, that cannot be the default, as it would make e.g. https://gitlab.freedesktop.org/mesa/mesa/container_registry unmanageable.

    • Yep, I know this is not for everyone so my intention was to hide it behind a a flip/switch/lever or an env variable

    Please register or sign in to reply
  • Michel Dänzer
    @daenzer started a thread
    • @bentiss wrote:

      • the variables part could be included in the build job, and we could probably add some defaults in ci-templates if this feels too much of a problem. OTOH, this are the most common set of variables the other projects require. Note that the DEBIAN_EXEC variable could be declared there too.

      IIUC what you mean, I happened to be working on something like this for Mesa: https://gitlab.freedesktop.org/mesa/mesa/merge_requests/2722

      @ivyl wrote:

      Even when you link to https://gitlab.freedesktop.org/wayland/ci-templates/blob/master/templates/debian.yml#L3 directly (which many project don't do),

      This is intentional at this point, as we still want to allow backwards incompatible changes to the templates.

      it feels more magical than having a simpler Dockerfile + a short bash script that handles conditional building available directly in your repo.

      For someone who isn't familiar with Docker already, tweaking a few variables and maybe a shell script seems more approachable to me.

      Also, there are pitfalls lurking with a "simple Dockerfile", which the templates are designed to avoid.

    Please register or sign in to reply
  • IIUC what you mean, I happened to be working on something like this for Mesa: https://gitlab.freedesktop.org/mesa/mesa/merge_requests/2722

    Yep sounds like it. I always used the global at the top level approach because when you deal with more than one distro, it's useful to point users to for the dependencies (for example: https://gitlab.freedesktop.org/libinput/libinput/blob/master/.gitlab-ci.yml#L65-73)

    Even when you link to https://gitlab.freedesktop.org/wayland/ci-templates/blob/master/templates/debian.yml#L3 directly (which many project don't do),

    This is intentional at this point, as we still want to allow backwards incompatible changes to the templates.

    We found a nicer way in libinput, instead of duplicating the shas: https://gitlab.freedesktop.org/libinput/libinput/blob/master/.gitlab-ci.yml#L27-44

    For someone who isn't familiar with Docker already, tweaking a few variables and maybe a shell script seems more approachable to me.

    Also, there are pitfalls lurking with a "simple Dockerfile", which the templates are designed to avoid.

    That was my main motivation to get away from Dockerfile. In theory, this is nice, but in practice, if you do not want to store all the garbage from your package manager, you have to write weird command like apt-get update; apt-get install foo ; apt-get clean for every package set you want to install.

Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment