README.md 6.14 KB
Newer Older
Daniel Stone's avatar
Daniel Stone committed
1
# freedesktop.org Helm deployment
Daniel Stone's avatar
Daniel Stone committed
2

Daniel Stone's avatar
Daniel Stone committed
3
4
5
6
7
8
9
10
11
This repository holds the configuration for the deployment of our GitLab service using Helm, as well as auxiliary services such as Grafana and the GitLab runners.

## Deploying GitLab itself

First, you will need access to the `fdo-gitlab` Kubernetes project, which you'll have if you're an admin.

Make sure you have this repository checked out, as well as the `helm-gitlab-omnibus` and `helm-gitlab-secrets` repositories. This repository holds the public master configuration; the Omnibus repository holds our fork of the GitLab Omnibus Helm chart which has been updated to support newer versions, as well as adapted to our deployment. This chart is no longer maintained upstream, but works well enough for us whilst we gradually move to GitLab's [cloud-native chart](https://docs.gitlab.com/charts/). The secrets repository contains passwords and API keys which cannot be made public.

Most configuration changes are made to `config.yaml` in this repository.
Daniel Stone's avatar
Daniel Stone committed
12
13
14
15
16

Check that you can see the running services:

```
$ kubectl get deployments
Daniel Stone's avatar
Daniel Stone committed
17
18
19
20
21
22
23
24
25
26
NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
gitlab-prod-cainjector                      1/1     1            1           4d4h
gitlab-prod-cert-manager                    1/1     1            1           4d4h
gitlab-prod-gitlab                          1/1     1            1           524d
gitlab-prod-gitlab-postgresql               1/1     1            1           524d
gitlab-prod-gitlab-redis                    1/1     1            1           524d
gitlab-prod-grafana                         1/1     1            1           4d
gitlab-prod-nginx-ingress-controller        3/3     3            3           4d4h
gitlab-prod-nginx-ingress-default-backend   2/2     2            2           4d4h
gitlab-prod-prometheus-server               1/1     1            1           4d1h
Daniel Stone's avatar
Daniel Stone committed
27
28
```

Daniel Stone's avatar
Daniel Stone committed
29
Change into the Omnibus chart directory, make any changes, and check the changes with a dry run:
Daniel Stone's avatar
Daniel Stone committed
30
31

```
Daniel Stone's avatar
Daniel Stone committed
32
$ cd ../helm-gitlab-omnibus
33
$ helm upgrade --dry-run -f ../helm-gitlab-config/config.yaml -f ../helm-gitlab-secrets/secrets.yaml gitlab-prod .
Daniel Stone's avatar
Daniel Stone committed
34
35
```

Daniel Stone's avatar
Daniel Stone committed
36
This will spew out the generated Kubernetes chart, which you can double-check. If you're happy with it, you can deploy for real by removing `--dry-run`.
Daniel Stone's avatar
Daniel Stone committed
37
38
39
40
41
42

Once it's up, wait for the instance to come back:
```
$ kubectl get deployment -w gitlab-prod-gitlab
```

Daniel Stone's avatar
Daniel Stone committed
43
44
45
46
47
48
49
50
51
52
You'll see AVAILABLE as 0 whilst GitLab restarts, then come back up as 1 when it's started.

You can run `kubectl get pods` to discover the name of the K8s pod that GitLab is running in. Once you have the name of the pod, you can run various commands in it, such as looking at the logs: `kubectl exec gitlab-prod-gitlab-307067958-z7s47 -c gitlab -it gitlab-ctl tail`

or opening a Ruby console and destroying as much data as possible: `kubectl exec gitlab-prod-gitlab-307067958-z7s47 -c gitlab -it gitlab-rails console production`


## Deploying GitLab runners

We currently have two fleets of GitLab runners - one hosted on [Packet](https://www.packet.com) and administered by us, and another hosted on [Hetzner](https://www.hetzner.com) and administered by the GStreamer Foundation.
Daniel Stone's avatar
Daniel Stone committed
53

Daniel Stone's avatar
Daniel Stone committed
54
The configuration and deployment of the Packet runners is automated and maintained in this repository.
Daniel Stone's avatar
Daniel Stone committed
55

Daniel Stone's avatar
Daniel Stone committed
56
Runners are created with `gitlab-runner-provision/gitlab-runner-packet.sh`, which requires environment variables to be set for the Packet project ID and API key, as well as the gitlab.fd.o shared-runner registration token. This script will generate a cloud-init YAML file to provision the runner, create a new server instance with Packet, and deploy it.
Daniel Stone's avatar
Daniel Stone committed
57

Daniel Stone's avatar
Daniel Stone committed
58
The runners should mostly not be reconfigured on the fly with SSH, but instead deleted and reprovisioned.
Daniel Stone's avatar
Daniel Stone committed
59
60


61
62
63
64
65
## Scripts for backup and re-deployment

The `gcp-scripts` folder holds a number of scripts used to create new Kubernetes clusters, imaging and attaching the existing volumes used for data storage, and allowing them to be copied between clusters. This is mostly only used when recreating a Kubernetes cluster, which is not something we have done since GKE has got better support for dynamically changing cluster attributes. Nevertheless, they're probably still useful to someone somewhere, particularly if you want to set up a new test cluster instance with the existing data.


Daniel Stone's avatar
Daniel Stone committed
66
## Creating a new GitLab deployment
Daniel Stone's avatar
Daniel Stone committed
67
68
69

Something bad has happened! Time to rebuild from scratch. :(

Daniel Stone's avatar
Daniel Stone committed
70
First, and this is **important**, make sure you manually zap the liveness/readiness checks inside the Helm chart. If you don't do this, when you get to the 'stop GitLab' point, K8s will zap the pod (because the service has failed) and restart it. This is not what you want.
Daniel Stone's avatar
Daniel Stone committed
71
72
73
74
75
76
77

Anyway.

Install the Helm chart from scratch:

```
$ cd helm-gitlab-omnibus
78
$ helm install -f ../helm-gitlab-config/config.yaml -f ../helm-gitlab-secrets/secrets.yaml --name gitlab-prod .
Daniel Stone's avatar
Daniel Stone committed
79
80
81
82
83
84
85
86
87
88
89
90
91
92
```

Wait for the deployment to go live:

```
$ kubectl get deployment -w gitlab-prod-gitlab
```

Find out its pod name, and get yourself a shell:
```
$ kubectl get pods
$ kubectl exec gitlab-prod-gitlab-307067958-z7s47 -c gitlab -it /bin/bash
```

Daniel Stone's avatar
Daniel Stone committed
93
Pull `/etc/gitlab/gitlab-secrets.json` from the `helm-config-secrets` repo manually, and pull the latest backup from Google storage. When you have these locally, push them into the pod:
Daniel Stone's avatar
Daniel Stone committed
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
```
$ kubectl cp gitlab-secrets.json gitlab-prod-gitlab-65c89cc6b7-bmgmx:/tmp/ -c gitlab
$ kubectl cp ~/tmp/1520868391_2018_03_12_10.5.4_gitlab_backup.tar gitlab-prod-gitlab-65c89cc6b7-bmgmx:/tmp/ -c gitlab
```

```
$ gitlab-ctl stop unicorn
$ gitlab-ctl stop sidekiq
$ gitlab-ctl status # make sure they're stopped
$ cp 123456789_2018_01_01_10.6.0_gitlab_backup.tar /var/opt/gitlab/backups/
$ cp gitlab-secrets.json /etc/gitlab/
$ gitlab-rake gitlab:backup:restore BACKUP=123456789_2018_01_01_10.6.0
$ gitlab-ctl reconfigure
$ gitlab-ctl restart
$ gitlab-rake gitlab:check SANITIZE=true
```

Daniel Stone's avatar
Daniel Stone committed
111
If you're fortunate, this has in fact worked. At this point, you can re-enable the readiness/liveness checks, push this with `helm upgrade` as per above, and cross your fingers that you've saved the day.
Daniel Stone's avatar
Daniel Stone committed
112

Daniel Stone's avatar
Daniel Stone committed
113
This process has in fact worked once before, so it's probably fine. For backup caveats, note https://gitlab.com/charts/charts.gitlab.io/issues/96
Daniel Stone's avatar
Daniel Stone committed
114
115

Good luck!