[th/gitlab-c-tiers] add multiple stages/tiers for tests

Thomas Haller requested to merge th/gitlab-ci-tiers into main

Originates from !1595 (closed).

We have many test configurations (i.e. distros like fedora:37, debian:9). Almost all of them run manually triggered, because running them every time would be wasteful.

Still, even as we trigger those tests only seldom, whenever we trigger them all together, they consume still too many resources of the freedesktop.org gitlab infrastructure.

One possibility would be to just drop old distros (e.g. fedora:30). Which tests are setup in gitlab-ci is constantly refined and adjusted. So dropping some distros is not necessarily wrong and bound to happen eventually.

However, I also don't find it great to just disable tests that are still passing. If we want to avoid consuming too many resources, we can just choose not to run those tests. We don't need to enforce that by deleting tests. Once deleted, such a configuration cannot be tested anymore as it would be too cumbersome to recreate the setup manually.

Instead, introduce stages/tiers to clearer mark configuration that we should test even less frequently.

Note that it is still required from the developer to not trigger too many tests at once, to not monopolize the CI resources. The stages should make that clearer to see, but don't solve it. Deleting tests might solve it, but only if we delete a significant number of those tests, which seems not desirable.

Merge request reports