Maintaining the cloud-init YAML by hand was getting pretty old,
especially as we needed to inline some things like the docker-gc script.

Rather than doing that, rewrite our cloud-init script as a Python
generator, which will allow us to more easily parameterise in order to
support things like different Docker GC strategies, different
architectures and disk layouts, etc - and not have to manually inline
our scripts.

This is currently implemented as a giant dict which matches the output
YAML, to ease diffing, but will become more programmatic over later
commits.

Signed-off-by: Daniel Stone <daniels@collabora.com>

Daniel Stone authored Apr 19, 2020 and Benjamin Tissoires committed Apr 19, 2020

I don't know whether or not this is actually a good idea. On one hand,
if we exclude them, then we cut runtime down, which means that our
runners are more efficiently utilised, which means that we need fewer
runners to serve our jobs, which means we have a better chance of
hitting local cache.

On the other hand, the upstream images don't cost us anything to pull,
and are competing with space for local images.

I think the best thing to do would be to see the effect after a week or
so, and whether or not we have space for them without jettisoning lots
of local images.

Signed-off-by: Daniel Stone <daniels@collabora.com>

Daniel Stone authored Apr 19, 2020 and Benjamin Tissoires committed Apr 19, 2020

Based on just looking at 'docker images' output.

Signed-off-by: Daniel Stone <daniels@collabora.com>

Signed-off-by: Daniel Stone <daniels@collabora.com>

Signed-off-by: Daniel Stone <daniels@collabora.com>

Ignore local mounts like the global cache.

Signed-off-by: Daniel Stone <daniels@collabora.com>

Signed-off-by: Daniel Stone <daniels@collabora.com>

Daniel Stone authored Apr 06, 2020 and Daniel Stone committed Apr 19, 2020

cf. https://about.gitlab.com/releases/2020/03/30/gpg-key-for-gitlab-package-repositories-metadata-changing/



Signed-off-by: Daniel Stone <daniels@collabora.com>

The controller all the time removes and adds the following:

```
ssl_trusted_certificate  /etc/ingress-controller/ssl/default-gitlab-pages-tls-full-chain.pem;
ssl_stapling             on;
ssl_stapling_verify      on;
```

It seems the CN doesn't need this as there is a job creating
the secrets on upgrade. But for us, we need it or the ssh
port will not be forwarded.

Signed-off-by: Benjamin Tissoires <benjamin.tissoires@gmail.com>

For the new charts to work, we need to namespace all of the config
files because we are using the root of the directory, not just
`./charts/gitlab-omnibus`.

This allows to be more compatible with the gitlab native kubernetes
chart and should allow us to move forward.

Signed-off-by: Benjamin Tissoires <benjamin.tissoires@gmail.com>

Signed-off-by: Benjamin Tissoires <benjamin.tissoires@redhat.com>

To simplify the calls to get_correct_iso_format()

Then:
`%s/datetime.datetime.fromisoformat(get_correct_iso_format(\(.*\)))/docker_fromisoformat(\1)/g`

Signed-off-by: Benjamin Tissoires <benjamin.tissoires@redhat.com>

Some images are not part of the db, either the db is not up to date,
either they were pulled manually. Ignore them.

Signed-off-by: Benjamin Tissoires <benjamin.tissoires@redhat.com>

so that we do not remove objects that have been created/used less than
a day ago.

Signed-off-by: Benjamin Tissoires <benjamin.tissoires@redhat.com>

Now that the history is offloaded to the gitlab server, we can start
we accurate data.

Signed-off-by: Benjamin Tissoires <benjamin.tissoires@redhat.com>

This should remove the complex need to listen to events, not have
history and whatnot.

Right now it is very simple, it counts since the beginning of the
available files in fdo-containers-usage.

Signed-off-by: Benjamin Tissoires <benjamin.tissoires@redhat.com>

Signed-off-by: Benjamin Tissoires <benjamin.tissoires@redhat.com>

Signed-off-by: Benjamin Tissoires <benjamin.tissoires@redhat.com>

problem is that the stored docker events are not that many, and
we mostly only get events from gitlab/gitlab-runner-helper

Signed-off-by: Benjamin Tissoires <benjamin.tissoires@redhat.com>

We want to listen to events, so clearing up our state at every run
will make this moot.

Signed-off-by: Benjamin Tissoires <benjamin.tissoires@redhat.com>

Signed-off-by: Benjamin Tissoires <benjamin.tissoires@redhat.com>

Signed-off-by: Benjamin Tissoires <benjamin.tissoires@redhat.com>

Signed-off-by: Benjamin Tissoires <benjamin.tissoires@redhat.com>

this makes easier to sort the list of last used images

Signed-off-by: Benjamin Tissoires <benjamin.tissoires@redhat.com>

Signed-off-by: Benjamin Tissoires <benjamin.tissoires@redhat.com>

otherwise:

  File "/usr/lib/python3/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.NotFound: 404 Client Error: Not Found ("No such container: 03c5217472db4bc979a167371262ce56e93dc2a9fd7e19791fa59ecf4cb0ef24")


Signed-off-by: Benjamin Tissoires <benjamin.tissoires@redhat.com>

We need to enforce working with UTC, otherwise we won't get accurate
expired time stamps.

Signed-off-by: Benjamin Tissoires <benjamin.tissoires@redhat.com>

Signed-off-by: Benjamin Tissoires <benjamin.tissoires@redhat.com>

Signed-off-by: Benjamin Tissoires <benjamin.tissoires@redhat.com>

Signed-off-by: Benjamin Tissoires <benjamin.tissoires@redhat.com>

As a reminder to myself in the future, this was generated with:

kubectl get pods --namespace marge-bot -o yaml > marge-mesa-config.yaml

Benjamin Tissoires authored Mar 27, 2020 and Benjamin Tissoires committed Mar 28, 2020

allows to run the script just once or in a loop

Signed-off-by: Benjamin Tissoires <benjamin.tissoires@gmail.com>

Benjamin Tissoires authored Mar 27, 2020 and Benjamin Tissoires committed Mar 28, 2020

Signed-off-by: Benjamin Tissoires <benjamin.tissoires@gmail.com>

Benjamin Tissoires authored Mar 27, 2020 and Benjamin Tissoires committed Mar 28, 2020

Signed-off-by: Benjamin Tissoires <benjamin.tissoires@gmail.com>

Benjamin Tissoires authored Mar 27, 2020 and Benjamin Tissoires committed Mar 28, 2020

Signed-off-by: Benjamin Tissoires <benjamin.tissoires@gmail.com>