ci-templates issueshttps://gitlab.freedesktop.org/freedesktop/ci-templates/-/issues2024-03-23T10:25:25Zhttps://gitlab.freedesktop.org/freedesktop/ci-templates/-/issues/69The ci-fairy image should allow pip without a venv2024-03-23T10:25:25ZPeter HuttererThe ci-fairy image should allow pip without a venvThe ci-fairy is the most basic image we provide but it does provide pip. During our install process (see `cbuild`) we remove `EXTERNALLY-MANAGED` before installing our packages (including `ci-fairy` itself) but then we restore that again...The ci-fairy is the most basic image we provide but it does provide pip. During our install process (see `cbuild`) we remove `EXTERNALLY-MANAGED` before installing our packages (including `ci-fairy` itself) but then we restore that again so the final image is still externally managed.
Arguably for the ci-fairy image we should remove this file altogether - the uses for this image is usually because it's small and installing a single package (e.g. black, ruff, pre-commit) is convenient. But they require venvs now which makes the lot a bit more complicated.
This isn't as trivial as I hoped though, `FDO_DISTRIBUTION_EXEC` is called before we restore the files so we need some command to signal `cbuild` that we don't want those restored.https://gitlab.freedesktop.org/freedesktop/ci-templates/-/issues/67Evaluate the new FreeBSD docker containers to replace the qemu solution2024-03-21T08:45:11ZEric Engestromeric@engestrom.chEvaluate the new FreeBSD docker containers to replace the qemu solutionSince gitlab-runner 16.9 (just released) + podman 5.0 (expected in the next couple of weeks), FreeBSD docker containers are supported (see https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/4551; the MR description has instruct...Since gitlab-runner 16.9 (just released) + podman 5.0 (expected in the next couple of weeks), FreeBSD docker containers are supported (see https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/4551; the MR description has instructions as well)
This could make all the `qemu`/`vmctl` stuff from https://gitlab.freedesktop.org/freedesktop/ci-templates/-/merge_requests/114 no longer necessary.https://gitlab.freedesktop.org/freedesktop/ci-templates/-/issues/66Create a default python venv in containers2024-01-31T11:56:25ZBenjamin TissoiresCreate a default python venv in containersPromoting one idea from @whot on !201 so it doesn't fall through the cracks:
thought bubble, I wonder if something like this in a default-provided `.bashrc` would be an option:
```
if command -v python3 > /dev/null ; then
python -m ...Promoting one idea from @whot on !201 so it doesn't fall through the cracks:
thought bubble, I wonder if something like this in a default-provided `.bashrc` would be an option:
```
if command -v python3 > /dev/null ; then
python -m venv /tmp/_venv
source /tmp/_venv/bin/activate
fi
```
i.e. our scripts always run in a venv. On the one hand this will make pip a lot more useful since the vast majority of our pip invocations will just work. OTOH for niche cases like testing different python versions or other specialised invocations it will make it very hard to work around this.
Optionally: `FDO_PYTHON_VENV: true` set at container-build time to drop that `.bashrc` in place.https://gitlab.freedesktop.org/freedesktop/ci-templates/-/issues/64How to explicitely specify arch for the image2024-01-08T13:45:46ZDavid HeidelbergHow to explicitely specify arch for the imageCurrently arch matches the arch of the runner. This is great for x86_64 -> x86_64 or aarch64 to aarch64.
For native armv7 or i386 there needs to be binding to aarch64 or x86_64, because we lack old i386 or armv7 servers.
For some reaso...Currently arch matches the arch of the runner. This is great for x86_64 -> x86_64 or aarch64 to aarch64.
For native armv7 or i386 there needs to be binding to aarch64 or x86_64, because we lack old i386 or armv7 servers.
For some reasons cross-compilations isn't also the best option (not everything is that well supported, and it introduce another complexity to the CI solutions)
For riscv64 we will have servers, but at current phase isn't viable to run infrastructure on RISC-V, so using x86-64 is also better choice.
Current option is to use something like:
Build images:
`FDO_BASE_IMAGE: "arm32v7/debian:$FDO_DISTRIBUTION_VERSION"`
Use:
```yaml
extends:
# - .fdo.distribution-image@debian
- .fdo.suffixed-image@debian # because ARMv7 & i386
```
but it isn't the best approach. After short discussion with @whot as a option would be introduce `FDO_FORCE_ARCH` which would specify custom ARCH.https://gitlab.freedesktop.org/freedesktop/ci-templates/-/issues/6332-bit templates running on 64-bit machines2023-12-25T19:31:03ZDavid Heidelberg32-bit templates running on 64-bit machinesWe cross-compile in `mesa/mesa` like:
- for x86_32 (i386) on x86_64 (Debian amd64),
- for armhf on arm64 (`aarch64` tag)
That is enough to get software compiled but not to prepare most of the software as a Debian package properly.
Sa...We cross-compile in `mesa/mesa` like:
- for x86_32 (i386) on x86_64 (Debian amd64),
- for armhf on arm64 (`aarch64` tag)
That is enough to get software compiled but not to prepare most of the software as a Debian package properly.
Sadly, the current state is that cross-compilation works only for a limited set of Debian packages.
I would see a solution to provide (at least) Debian templates with:
- armhf (can run on `aarch64` tagged runners natively)
- i386 (on x86_64).https://gitlab.freedesktop.org/freedesktop/ci-templates/-/issues/58ci-fairy: add check for FDO_DISTRIBUTION_TAG bump2023-02-15T22:07:30ZSimon Sercontact@emersion.frci-fairy: add check for FDO_DISTRIBUTION_TAG bumpIt would be nice if ci-fairy checked that `FDO_DISTRIBUTION_TAG` is correctly bumped when `.gitlab-ci.yml` is updated.It would be nice if ci-fairy checked that `FDO_DISTRIBUTION_TAG` is correctly bumped when `.gitlab-ci.yml` is updated.https://gitlab.freedesktop.org/freedesktop/ci-templates/-/issues/56[RFC] Switch from the vfs to overlayfs storage driver2023-02-02T09:52:35ZMartin Roukala[RFC] Switch from the vfs to overlayfs storage driverBuilding big containers using ci-templates is excruciatingly slow on SSD drives. I assume this is due to the storage driver being set to "vfs" rather than overlayfs (using fuse-overlayfs).
This seems to be due to https://gitlab.freedesk...Building big containers using ci-templates is excruciatingly slow on SSD drives. I assume this is due to the storage driver being set to "vfs" rather than overlayfs (using fuse-overlayfs).
This seems to be due to https://gitlab.freedesktop.org/freedesktop/ci-templates/-/blob/master/bootstrap/bootstrap_fedora.sh#L79, but it is also found in the qemu image: https://gitlab.freedesktop.org/freedesktop/ci-templates/-/blob/master/bootstrap/cbuild#L648https://gitlab.freedesktop.org/freedesktop/ci-templates/-/issues/54ci-fairy: allow use of private commit email address2023-03-28T09:58:23ZJake Daneci-fairy: allow use of private commit email addressGitLab provides an automatically generated private commit email address, so that I can keep my email address private. See [the docs](https://docs.gitlab.com/ee/user/profile/#use-an-automatically-generated-private-commit-email). The email...GitLab provides an automatically generated private commit email address, so that I can keep my email address private. See [the docs](https://docs.gitlab.com/ee/user/profile/#use-an-automatically-generated-private-commit-email). The email address has the form `<number>-<username>@users.noreply.<gitlab instance domain>`.
ci-fairy throws error "git author email invalid" when I use the private commit email address. See [here in the code](https://gitlab.freedesktop.org/freedesktop/ci-templates/-/blob/master/tools/ci_fairy.py#L203). I ask that this be reconsidered and private commit email addresses are allowed to be used. These are a standard GitLab function.
I'm not comfortable publicly disclosing this personal information and I don't think it's a price I should have to pay for wanting to contribute to FOSS projects.https://gitlab.freedesktop.org/freedesktop/ci-templates/-/issues/53Add new template with ANSI colors2022-11-25T08:28:16ZBenjamin TissoiresAdd new template with ANSI colorsThe following discussion from !105 should be addressed:
- [ ] @whot started a [discussion](https://gitlab.freedesktop.org/freedesktop/ci-templates/-/merge_requests/105#note_1535185):
> I reckon that (in a follow-up) we should add ...The following discussion from !105 should be addressed:
- [ ] @whot started a [discussion](https://gitlab.freedesktop.org/freedesktop/ci-templates/-/merge_requests/105#note_1535185):
> I reckon that (in a follow-up) we should add some `ci-templates/utils` template or somesuch that provides:
> ```yaml
> .ansi-colors:
> variables:
> RED: "\033[0;31m"
> ```
> etc so any job wanting to print with colors can just `extends` from that job.https://gitlab.freedesktop.org/freedesktop/ci-templates/-/issues/40Include multi-arch manifest into `cbuild`2021-07-30T14:07:44ZBenjamin TissoiresInclude multi-arch manifest into `cbuild`The following discussion from !105 should be addressed:
- [ ] @whot started a [discussion](https://gitlab.freedesktop.org/freedesktop/ci-templates/-/merge_requests/105#note_1007774): (+5 comments)
> are we at the point yet where t...The following discussion from !105 should be addressed:
- [ ] @whot started a [discussion](https://gitlab.freedesktop.org/freedesktop/ci-templates/-/merge_requests/105#note_1007774): (+5 comments)
> are we at the point yet where this would be better off in a shell script? :)https://gitlab.freedesktop.org/freedesktop/ci-templates/-/issues/26Non-merged merge requests push container images on the upstream projects cont...2020-12-07T03:31:16ZJonas ÅdahlNon-merged merge requests push container images on the upstream projects container registrySo in two scenarios I've seen container images being rebuilt and included in the 'upstream repo' container registry before the merge request was merged.
* It happens when the detached merge request pipeline was running on the upstream p...So in two scenarios I've seen container images being rebuilt and included in the 'upstream repo' container registry before the merge request was merged.
* It happens when the detached merge request pipeline was running on the upstream project, even though the merge request was opened from a fork.
* It happens when a merge request is run for a branch on the upstream repo itself. This is still a very common way of handling branches in GNOME.
This is quite annoying, for three reasons; one is that if there are two merge requests that happen to have the same tag, only one will actually build the image, the otherone will just see that it (the wrong one) already exist and bail.
Another annoyance is that one has to temporarly do `FDO_FORCE_REBUILD=1` if there was any changes, then wait for the build to continue, then push another update without the force-rebuild env var set.
The third annoyance is that the upstream container registry will contain "wip"-container images.
How could we solve this?
One idea would be to contain some semi random token in the container images when they are built, e.g. the hash of `HEAD` of the merge request branch. This way, when merging, the pipeline could use the hash and do a simple "copy" when finally merging, instead of rebuilding once again.
Any other ideas?https://gitlab.freedesktop.org/freedesktop/ci-templates/-/issues/19Arch-specific templates in distribution-image and suffixed-image?2020-11-13T09:37:38ZPeter HuttererArch-specific templates in distribution-image and suffixed-image?As pointed out by @daenzer in https://gitlab.freedesktop.org/freedesktop/ci-templates/-/merge_requests/47#note_688356
We have arch-specific `container-build` and `qemu-build` templates now, with aliases for the non-suffixed ones pointin...As pointed out by @daenzer in https://gitlab.freedesktop.org/freedesktop/ci-templates/-/merge_requests/47#note_688356
We have arch-specific `container-build` and `qemu-build` templates now, with aliases for the non-suffixed ones pointing to the `x86_64` ones. But our `distribution-image` and `suffixed-image` templates don't have any arch-specific versions. What's the plan there?
The only thing those templates right now would do is to add `tags: [ aarch64]`.
But that brings up another issue: the arch is only used for the `buildah` image but not encoded in the resulting image. So two `container-build` templates with different suffixes will end up pushing the same image to the registry - it falls on the user to distinguish those too by changing the tag or using a suffix.
So the most obvious use-case doesn't work properly:
```yaml
.fedora:
FDO_DISTRIBUTION_PACKAGES: '....'
FDO_DISTRIBUTION_TAG: '2020-11-11.0'
FDO_DISTRIBUTION_VERSION: '33'
build f33 x86:
extends:
- .fedora
- .fdo.container-build@fedora@x86_64
build f33 arm:
extends:
- .fedora
- .fdo.container-build@fedora@aarch6
run f33:
extends:
- .fedora
- .fdo.distribution-image@fedora
script:
- echo "I don't know which image I'm running on"
```
I think we need to encode the arch in the image name.
cc @jadahl, @bentiss, @daenzerhttps://gitlab.freedesktop.org/freedesktop/ci-templates/-/issues/14Enforcing new image tags to be rebuilt when merging an MR2020-07-20T13:53:21ZMichel DänzerEnforcing new image tags to be rebuilt when merging an MRIt's happened a few times now in Mesa that the post-merge CI pipeline got broken due to the pre-merge one using older images which could no longer be rebuilt (due to other changes being merged between when those images were built and whe...It's happened a few times now in Mesa that the post-merge CI pipeline got broken due to the pre-merge one using older images which could no longer be rebuilt (due to other changes being merged between when those images were built and when the MR was merged). Latest example: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5905
It would be nice to have some kind of mechanism to enforce in the pipeline before merging an MR that image tags which don't exist in the upstream registry yet are built from scratch. (Only if an existing downstream image is older than the target branch?)https://gitlab.freedesktop.org/freedesktop/ci-templates/-/issues/12Forked registry cleanup2023-02-14T00:29:36ZJordan PetridіsForked registry cleanupCurrently the workflow goes like this, build image `Foo` in `Upstream/foo` registry, forked projects check `Upstream/foo` and copy the image into their forked registry. This means that even if we cleanup the upstream repo the forked repo...Currently the workflow goes like this, build image `Foo` in `Upstream/foo` registry, forked projects check `Upstream/foo` and copy the image into their forked registry. This means that even if we cleanup the upstream repo the forked repositories will still have references to the images preventing them from being cleaned up.
I was looking at how gitlab is tagging their "cache" container/volumes the other day and noticed that they have a special `LABEL` which they are then able to filter for and purge them.
https://gitlab.com/gitlab-org/gitlab-runner/blob/master/packaging/root/usr/share/gitlab-runner/clear-docker-cache
```
CONTAINERS=$(docker ps -a -q \
--filter=status=exited \
--filter=status=dead \
--filter=label=com.gitlab.gitlab-runner.type=cache)
if [ -n "${CONTAINERS}" ]; then
docker rm -v ${CONTAINERS}
fi
```
While this won't work as is since we want to go through the Gitlab API, we could maybe could adapt it and workaround it. Instead of copying the image as is, we build a new one with an extra layer of `LABEL=org.freedesktop.registry.is_forked=true"` and push that. Then have a job that crawls the registry api and inspects the images for this layer, and if its not matching the current tag, delete it.
I can think of a couple edge cases though as is,
* Multiple stable images you want to keep, the `$TAG` will probably be different for stable branches
* Outdated checkouts that need rebasing will have an older tag(s), that might end up causing unintended deletions (though always repullable)https://gitlab.freedesktop.org/freedesktop/ci-templates/-/issues/10Don't build pages for branches other than master2020-09-11T12:37:04ZPeter HuttererDon't build pages for branches other than masterWe're currently building pages in any CI, so in theory a push to a branch that isn't master will overwrite our official documentation with that branch's documentation.
Not that much of a big deal for the main project since we rarely pus...We're currently building pages in any CI, so in theory a push to a branch that isn't master will overwrite our official documentation with that branch's documentation.
Not that much of a big deal for the main project since we rarely push non-master branches (we do sometimes though). But it is more of an issue in user branches. You can't use personal pages as a reference for a MR ("look, this is how it looks like") unless you stop pushing all other branches.
So we should probably have a filter for master only on the main pipeline and (maybe?) push a branch's pages into a subdirectory or something like that.