Backup and restore

README.md

@@ -43,3 +43,63 @@ in it, such as looking at the logs:
 
 or opening a Ruby console and destroying as much data as possible:
 `kubectl exec gitlab-prod-gitlab-307067958-z7s47 -c gitlab -it gitlab-rails console production`
+
+
+
+### Installing from scratch
+
+Something bad has happened! Time to rebuild from scratch. :(
+
+First, and this is **important**, make sure you manually zap the
+liveness/readiness checks inside the Helm chart. If you don't do this, when you
+get to the 'stop GitLab' point, K8s will zap the pod (because the service has
+failed) and restart it. This is not what you want.
+
+Anyway.
+
+Install the Helm chart from scratch:
+
+```
+$ cd helm-gitlab-omnibus
+$ helm install -f ../helm-gitlab-config/config.yaml -f ../helm-gitlab-secrets/secrets.yaml --name gitlab-prod ./charts/gitlab-omnibus
+```
+
+Wait for the deployment to go live:
+
+```
+$ kubectl get deployment -w gitlab-prod-gitlab
+```
+
+Find out its pod name, and get yourself a shell:
+```
+$ kubectl get pods
+$ kubectl exec gitlab-prod-gitlab-307067958-z7s47 -c gitlab -it /bin/bash
+```
+
+Pull `/etc/gitlab/gitlab-secrets.json` from the `helm-config-secrets` repo
+manually, and pull the latest backup from Google storage. When you have these
+locally, push them into the pod:
+```
+$ kubectl cp gitlab-secrets.json gitlab-prod-gitlab-65c89cc6b7-bmgmx:/tmp/ -c gitlab
+$ kubectl cp ~/tmp/1520868391_2018_03_12_10.5.4_gitlab_backup.tar gitlab-prod-gitlab-65c89cc6b7-bmgmx:/tmp/ -c gitlab
+```
+
+```
+$ gitlab-ctl stop unicorn
+$ gitlab-ctl stop sidekiq
+$ gitlab-ctl status # make sure they're stopped
+$ cp 123456789_2018_01_01_10.6.0_gitlab_backup.tar /var/opt/gitlab/backups/
+$ cp gitlab-secrets.json /etc/gitlab/
+$ gitlab-rake gitlab:backup:restore BACKUP=123456789_2018_01_01_10.6.0
+$ gitlab-ctl reconfigure
+$ gitlab-ctl restart
+$ gitlab-rake gitlab:check SANITIZE=true
+```
+
+If you're fortunate, this has in fact worked. At this point, you can re-enable
+the readiness/liveness checks, push this with `helm upgrade` as per above, and
+cross your fingers that you've saved the day.
+
+This process has in fact worked once before, so it's probably fine.
+
+Good luck!