Restrict runners to valid freedesktop projects only, v2

Over the first week of June 2022, we have seen several attempts at mining cryptocurrencies on our runners.

And on March 12 2023, the same hackers came back and went one step further compromising our entire fleet of runners. So we have to take actions.

While we are an open community and try to provide a great free infrastructure for open source development, those attempts are clearly a breach in the trust we give to our users.

This is why we are sad to announce that we have to take actions to restrict how the runners are freely available to anybody.

This restrictions are coming on top of the automated detection and prevention of this kind of use, of course.

What it means for me, a regular user and contributor to gitlab.freedesktop.org?

Starting `2023-03-14`, there is a new banner added in each job log which will tell you if you'll keep your privileges or not:

Checking if the user of the pipeline is allowed...
Thank you for contributing to freedesktop.org

Checking if the user of the pipeline is allowed...
Checking if the job's project is part of a well-known group...
Checking if the job is part of an official MR pipeline...
***************************************************************************************************************************
Looks like you don't have enough privileges, please see https://gitlab.freedesktop.org/freedesktop/freedesktop/-/issues/XXX
***************************************************************************************************************************
Skipping permission check for now

The policy is still not enforced, so both outputs will not prevent the job to be run, but this is a good way to see if you are impacted or not.

We are in the testing period, and enforcing it right now might be disastrous for some projects and for the admins lives.

Starting `2023-03-23`, or before, if needed, the above policy will be enforced.

Projects with MR detached pipelines will still be able to be run for random contributors, without manual intervention.

In other words. You won't be able to run CI here if you currently see:

***************************************************************************************************************************
Looks like you don't have enough privileges, please see https://gitlab.freedesktop.org/freedesktop/freedesktop/-/issues/XXX
***************************************************************************************************************************

If you think this is a mistake please reach out to us.

Starting `2023-04-16`, or before, if needed, only project members will be able to run CI

So you won't be able to run CI here if you currently see:

Checking if the user of the pipeline is allowed...
Checking if the job's project is part of a well-known group...
Checking if the job is part of an official MR pipeline...
Thank you for contributing to freedesktop.org

In other words, if the third check here (Checking if the job is part of an official MR pipeline...) is present, you won't be able to run CI yourself.

If you are allowed to push or merge on the target branch in the repository

There is likely a problem in the protected branches configuration.

What we have seen is that some projects are protecting all the branches with a wildcard * as nopush/nomerge. This makes gitlab believe that nobody can push on the repo when creating the MR pipelines (yes it's a bug, someone please file it or solve it).

Or it could also simply be that the target project is not using detached MR pipelines, meaning that direct project members might not be able to run CI if they are not part of an official group.

If you are not allowed to push or merge on the target branch in the repository

Well, you'll have to ask for one of the maintainers/developers of the project you want to contribute to to run the pipeline for you. They just have to go to the MR, click on the Pipeline tab, and hit Run pipeline

Starting `[date-to-be-determined-if-there-are-abuse]`, or before, if needed, only project members with enough privileges will be able to run CI

Currently we require the Guest level access to be able to run CI. If anything happens, we can still restrict the required level to be Reporter, Developer or above only.

Will there be exceptions?

Yes and no. The current implementation makes it possible to have exception:

The list of valid users is the one we get from the CI-OK group.

This group is internal, and noone should need to be manually added to it, because it's a meta-group of the other groups.

So if you are not allowed to run CI on our shared runners, please apply for the 'guest' role in the project you are contributing too, this should automatically give you CI access.

But I just want to run `[replace-me-with-some-random-project]` in my own namespace...

Sorry. This is the only way forward we can see.

If the project you want to use our sponsored runners is freedesktop related, why not contribute?

If the project is not, well, we can not actually let you run your tasks on our shared runners. Just disable the shared runners in your project settings and bring in your owns!

How are you deciding who gets to run what?

The first source of trust is the CI-OK group.

In theory, this should be sufficient as long as all groups are added to it and that you have basic access to any of the official project.

It's everyone responsibility to add a user to an official group. Please think twice when doing so, you might give cryptominers the keys to our kingdom

However, there is a gitlab bug where the REST API for querying a group member might not return a valid user while the user clearly sees that it is part of the group in its profile page.

For these cases, we implement the following workaround:

if you project implements detached MR pipelines,
and if the user has the right to push or merge in the target branch

Then the CI will be authorized.

If the user doesn't have the rights but detached MR pipelines are enabled, the CI will temporary pass (see the date above for when this privilege will be lifted).

What it means for me, a maintainer of a project part of gitlab.freedesktop.org?

Hopefully nothing. Contributors should still be able to run CI on the MRs, and you can still have your project being tested as previously.

For drive-by contributors, they may not have access to the CI, and someone with CI-OK privileges will have to got to the MR, got to the pipeline tab, then hit the Run Pipeline button. This will create a pipeline with the correct privileges, and you are good to continue.

If you don't want to have developers hit that button, we have a temporary workaround. (It is temporary because we might decide to remove it for all non project contributors).

So, what you may want to have are detached MR pipelines. For that, just append the following snippet to your .gitlab-ci.yaml:

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
    - if: $CI_PIPELINE_SOURCE == 'push'

Yes, this will create duplicated pipelines, one for regular push, and one for MR. This is expected and you can talk to us on #freedesktop over IRC if you want to enhance the situation.

The benefits of having detached pipeline is that any user that creates a MR will be able to run the CI. Note that currently a non project member can run the CI with that workaround. In one month from now, only project members will get the policy validated for them. Drive-by contributors will require a manual action from a project member.

What it means for me, a maintainer of a shared runner of gitlab.freedesktop.org?

Every shared runner will have to use the following pre-get_sources and pre_build scripts in the [[runners]] section:

[[runners]]
  name = "xxx"
  url = "https://gitlab.freedesktop.org"
  executor = "docker"
  ...
  
  pre_get_sources_script = "/host/bin/curl --cacert /host/ca-certificates.crt -s -L --retry 4 -f --retry-delay 60 https://gitlab.freedesktop.org/freedesktop/helm-gitlab-infra/-/raw/main/runner-gating/runner-gating.sh | sh -s -- pre_get_sources_script"
  pre_build_script = "/host/bin/curl -s -L --cacert /host/ca-certificates.crt --retry 4 -f --retry-delay 60 https://gitlab.freedesktop.org/freedesktop/helm-gitlab-infra/-/raw/main/runner-gating/runner-gating.sh | sh"
  [runners.docker]
    ...

This script will prevent the use of the runner outside of its legitimate usage, and will also allow us, sysadmins to change the rules on the fly without have to ask everybody to re-apply changes.

We have to apply it twice because if GIT_POLICY is set to None, the pre_get_sources_script is not run. And some projects use it through $CI_PRE_CLONE_SCRIPT. So it would be a pretty big hole to not check the policy there.

The binaries in /host/bin/ come from a :ro volume mount from the host, and we currently need {ca-certificates.crt, /bin/curl, /bin/jq to be available in the mount. jq and curl are statically compiled and provided through this package registry.

This volume is mounted as such:

    volumes = ["/var/host:/host:ro"]

The pre_get_sources_script runs inside the gitlab helper image which is also set by the runner config so its a trusted source.

The pre_build_script is executed in the job's container, just before the before_script. So no assomption can be done on the environment, besides the availability of sh.

Edited Mar 17, 2023 by Benjamin Tissoires

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information