Commit bbaac055 authored by Daniel Stone's avatar Daniel Stone
Browse files

Backup and restore

parent 9f48112f
......@@ -43,3 +43,63 @@ in it, such as looking at the logs:
or opening a Ruby console and destroying as much data as possible:
`kubectl exec gitlab-prod-gitlab-307067958-z7s47 -c gitlab -it gitlab-rails console production`
### Installing from scratch
Something bad has happened! Time to rebuild from scratch. :(
First, and this is **important**, make sure you manually zap the
liveness/readiness checks inside the Helm chart. If you don't do this, when you
get to the 'stop GitLab' point, K8s will zap the pod (because the service has
failed) and restart it. This is not what you want.
Install the Helm chart from scratch:
$ cd helm-gitlab-omnibus
$ helm install -f ../helm-gitlab-config/config.yaml -f ../helm-gitlab-secrets/secrets.yaml --name gitlab-prod ./charts/gitlab-omnibus
Wait for the deployment to go live:
$ kubectl get deployment -w gitlab-prod-gitlab
Find out its pod name, and get yourself a shell:
$ kubectl get pods
$ kubectl exec gitlab-prod-gitlab-307067958-z7s47 -c gitlab -it /bin/bash
Pull `/etc/gitlab/gitlab-secrets.json` from the `helm-config-secrets` repo
manually, and pull the latest backup from Google storage. When you have these
locally, push them into the pod:
$ kubectl cp gitlab-secrets.json gitlab-prod-gitlab-65c89cc6b7-bmgmx:/tmp/ -c gitlab
$ kubectl cp ~/tmp/1520868391_2018_03_12_10.5.4_gitlab_backup.tar gitlab-prod-gitlab-65c89cc6b7-bmgmx:/tmp/ -c gitlab
$ gitlab-ctl stop unicorn
$ gitlab-ctl stop sidekiq
$ gitlab-ctl status # make sure they're stopped
$ cp 123456789_2018_01_01_10.6.0_gitlab_backup.tar /var/opt/gitlab/backups/
$ cp gitlab-secrets.json /etc/gitlab/
$ gitlab-rake gitlab:backup:restore BACKUP=123456789_2018_01_01_10.6.0
$ gitlab-ctl reconfigure
$ gitlab-ctl restart
$ gitlab-rake gitlab:check SANITIZE=true
If you're fortunate, this has in fact worked. At this point, you can re-enable
the readiness/liveness checks, push this with `helm upgrade` as per above, and
cross your fingers that you've saved the day.
This process has in fact worked once before, so it's probably fine.
Good luck!
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment