Due to an influx of spam, we have had to impose restrictions on new accounts. Please see this wiki page for instructions on how to get full permissions. Sorry for the inconvenience.
Admin message
Equinix is shutting down its operations with us on April 30, 2025. They have graciously supported us for almost 5 years, but all good things come to an end. We are expecting to transition to new infrastructure between late March and mid-April. We do not yet have a firm timeline for this, but it will involve (probably multiple) periods of downtime as we move our services whilst also changing them to be faster and more responsive. Any updates will be posted in freedesktop/freedesktop#2011 as it becomes clear, and any downtime will be announced with further broadcast messages.
Enforcing new image tags to be rebuilt when merging an MR
It's happened a few times now in Mesa that the post-merge CI pipeline got broken due to the pre-merge one using older images which could no longer be rebuilt (due to other changes being merged between when those images were built and when the MR was merged). Latest example: mesa/mesa!5905 (merged)
It would be nice to have some kind of mechanism to enforce in the pipeline before merging an MR that image tags which don't exist in the upstream registry yet are built from scratch. (Only if an existing downstream image is older than the target branch?)
Designs
Child items
...
Show closed items
Linked items
0
Link issues together to show that they're related.
Learn more.
checking whether an image exists in FDO_UPSTREAM_REPO is easy
checking if the downstream image is older than the target branch? possible through the API (ci-fairy maybe?)
some kind of mechanism to enforce this? That's tricky.
CI_JOB_MANUAL is set if the job is manually started, so we could hook this up to FDO_FORCE_REBUILD but you'd still have to re-run every job, I don't think there's a pipeline-wide option here.
An option could be to have a manual job that pokes the API to run the pipeline and has FDO_FORCE_REBUILD set. Problem is that job would complete successfully (it only pokes the pipeline) and the pipeline result wouldn't be tied to the MR, I think. Plus, I don't know if the normal job tokens give you permission to do this.
checking if the downstream image is older than the target branch? possible through the API (ci-fairy maybe?)
Does the API allow determining when the branch was actually updated (as opposed to the commit date of its HEAD)?
Another possibility might be to check if the target branch HEAD commit is part of the history of the commit which built the local image (fdo.commit label). If it's not, an image rebuild is forced.
Does the API allow determining when the branch was actually updated (as opposed to the commit date of its HEAD)?
hmm, no. But we can check for the latest pipeline with that sha and its finished_at, that's close enough.
It would have to be automated to be useful.
Does marge-bot have an option to trigger a custom pipeline before merging? That would solve your case - marge decides this MR will get merged, triggers a detached pipeline with FDO_FORCE_REBUILD=1 and merges upon completion of that pipeline.
Hmm, for a proof of concept, maybe Mesa could set FDO_FORCE_REBUILD=1 in before_script if the job was triggered by Marge, and the tag doesn't exist in the upstream registry. If that works well, maybe we can come up with some kind of simpler mechanism in the templates.
I think I just understood the issue (please correct me if I am wrong):
Person A opens a MR bumping the CI images and doing some changes in it
Person B merges upstream changes in the CI images that would break the CI changes from person A
when we do the pre-merge pipeline (with a rebase on top of master), the pipeline succeeds because it uses the images built in person A namespace the first time the MR was opened/pushed
once the changes are pushed upstream, the pipeline tries to rebuilt the new image and fail.
The one thing I do not understand, is why this even happen in the first place. When the pre-merge pipeline is run, it is subsequent to a rebase on top of master. This rebase can't be without conflict because of the changes introduced by person B. So there must be at least a conflict resolution for the FDO_DISTRIBUTION_TAG. When that happens, we should not re-use the previous tag, but use a new one (matching the date) to force rebuilding the image.
IMO, this can't be solved at the ci-templates level, but is more of a social problem, because it will depend on so many cases that we can't do anything smart. The simplest solution would be to have FDO_FORCE_REBUILD = <namespace> != FDO_UPSTREAM_REPO (in pseudo code), but that means that we would rebuild the images for every push, defeating the ci-templates benefits.
Another possibility might be to check if the target branch HEAD commit is part of the history of the commit which built the local image (fdo.commit label). If it's not, an image rebuild is forced.
This is more elegant, but will likely result in the same drawbacks than forcing the rebuild at every push not on master :(
So there must be at least a conflict resolution for the FDO_DISTRIBUTION_TAG.
In the last Mesa case above, the "person B" MR didn't bump the tag for some of the images affected by it. It's an easy mistake to make, and easy to miss during review.
Even if there is a conflict on the tag though, the same tag which was used since the "person A" MR was created could be a valid resolution of the conflict at least in theory and some cases. Again, easy to miss during review (I forgot about this risk when reviewing the "person A" MR again before merging, even though I had thought about this before; most reviewers likely wouldn't be aware of the risk).
This is more elegant, but will likely result in the same drawbacks than forcing the rebuild at every push not on master :(
It would mean images for which the tag is bumped in an MR are rebuilt after every rebase. IMHO that's a price worth paying for preventing this kind of issue. (Particularly for Mesa, which normally only actually runs the pipeline once before merging an MR; other projects are welcome to emulate that :)
I am not saying we shouldn't do any automatic check, but it doesn't seem to me that a generic solution at the ci-templates level can be found that will work for every users/workflow.
I'd rather fix the problem in mesa (see below), fix it in the other projects that encounter this issue, and then think of a generic solution here. My point is that a ci-templates solution only will enforce every users a particular workflow when for many projects this is not as bad as it can be on the mesa project (because of exposure, number of people, etc...)
It would mean images for which the tag is bumped in an MR are rebuilt after every rebase. IMHO that's a price worth paying for preventing this kind of issue. (Particularly for Mesa, which normally only actually runs the pipeline once before merging an MR; other projects are welcome to emulate that :)
Looking at the numbers since end of June, beginning of July, and we had far too much registry egress. I thought it was because gstreamer updated their base image, but it seems that there is a mix of everything too. So this behavior will just add more noise and costs for little benefits.
A solution could be to add a before_script in the mesa containers CI, that uses what you and @whot suggested: if $GITLAB_USER_LOGIN is margebot, and if fdo.commit is not in the history, set FDO_FORCE_REBUILD=1.
This way, we fix the mesa issue, and document it. When other comes in we can always suggest that, and eventually decide if we can turn this into a generic implementation.
If the templates were to enforce that a new image in the forked registry is up to date when merging an MR, the post-merge pipeline in the main project could simply copy the image from there instead of rebuilding it again. This would make a pretty big difference for Mesa, where some of the image builds are taking tens of minutes.
Edit: Assuming it's possible to determine in the post-merge pipeline which project the MR was merged from... not sure how offhand.
well, doesn't that defeat the security side of ci-templates? Anybody can then upload a buggy image on his own fork, ensuring it works with the MR, but also add bad things in it? The mesa pipeline would then happily clone it, and we will start to run untrusted images.
Maybe a solution would be to have the pre-merge pipeline run in the mesa namespace so marge-bot can push images to the mesa registry. No?
Yeah, I think the key is 'enforce that a new image in the forked registry is up to date when merging an MR', i.e. that Marge executing would always run the downstream container build no matter what, then a post-merge pipeline would copy from downstream to upstream.