Migration of the DB to packet-HA (tentative: June 13, 2021)
Cc: @daniels
The new cluster (with High Availability control plane) on packet is already handling the gitlab workload. This issue is what need to be done during the last migration step: the db move.
prep step
-
all the git repos must have been migrated from no-replicas
shard togitaly-[0-2]
-
actual git repos -
snippets git repos
-
-
mark gitaly-[0-2]
to be the only one receiving new repos (in Repository Storage) -
remove no-replicas
anddefault
shards from the old cluster -
default
is now a link togitaly-0
but shouldn't be used for storage
step 0: notify users
Few days before the upgrade, we need to notify users that there's going to be a day where we take the service down.
current tentative date is June 13, 2021
-
banner created and active
prevent writes to the current DB
Of course, we don't want to migrate an almost accurate DB and leave the last work behind
-
scale down webservice pods on packet-HA -
scale down gitaly pods on packet-HA -
scale down gitlab-exporter on packet-HA -
scale down gitlab-pages on packet-HA - note: maybe optional
-
scale down registry on packet-HA -
scale down sidekiq pods on packet-HA -
scale down any task-runner-backup pods on packet-HA -
run the Redis clear job on packet -
scale down registry on packet -
scale down gitlab-pages on packet -
scale down gitlab-exporter on packet
backup
option a: run backup from velero
On the old cluster:
-
velero backup create postgresql-2021-06-13-HH-MM --include-namespaces gitlab --selector app=postgresql --default-volumes-to-restic
-
velero backup create redis-2021-06-13-HH-MM --include-namespaces gitlab --selector app=redis --default-volumes-to-restic
-
wait for both backups to finish
option b: run the backup script (on packet-HA)
https://docs.gitlab.com/charts/installation/upgrade.html tells us to use https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/scripts/database-upgrade
-
curl -s "https://gitlab.com/gitlab-org/charts/gitlab/raw/v4.12.3/scripts/database-upgrade" | bash -s pre
setup the new cluster to handle the db on the new cluster itself
-
configs/packet-HA/globals.gotmpl
: changeglobal.psql.host: gitlab-prod-postgresql-postgresql
(can we modify the redis host without its data? -> yes)
-
configs/packet-HA/globals.gotmpl
: changeglobal.redis.host: gitlab-prod-redis-master
Note: do not deploy the configs or the pods will be scaled up again
task-runner
pod on packet (only required in option b)
spin up the -
manually edit gitlab-prod-task-runner
cm on packet-HA-
kubectl -n gitlab edit cm gitlab-prod-task-runner
-
database.yml.erb
needs to be updated
-
-
delete task-runner pod on packet-HA
restore the DB on packet
option a: velero
On the new cluster:
-
remove redis deployment manually -
remove postgresql deployment manually -
remove postgresql PVC manually -
remove redis PVC manually -
remove postgresql PV manually (optional) -
remove redis PV manually (optional) -
velero backup describe postgresql-2021-06-13-HH-MM
-
Phase: Completed
-
-
velero backup describe redis-2021-06-13-HH-MM
-
Phase: Completed
-
-
velero restore create --from-backup postgresql-2021-06-13-HH-MM
-
velero restore create --from-backup redis-2021-06-13-HH-MM
option b: manual backup/restore
-
curl -s "https://gitlab.com/gitlab-org/charts/gitlab/raw/v4.12.3/scripts/database-upgrade" | bash -s post
Create a temp ip for the tests only
-
kubectl -n gitlab delete svc fdo-gitlab-ingress-nginx-prod-controller
-
edit configs/packet-HA/fdo-gitlab-ingress-nginx.gotmpl
and comment outloadBalancerIP:
-
helmfile -e packet-HA -l chart=ingress-nginx -i apply
re-deploy the chart on packet-HA
-
helmfile -e packet-HA sync
tests (with the temp IP)
-
kubectl -n gitlab get svc fdo-gitlab-ingress-nginx-prod-controller
-
gitlab.freedesktop.org
should work -
*.pages.freedesktop.org
should still be OK -
regular git pull works -
regular git push works
reset the published ip
-
kubectl -n gitlab delete svc fdo-gitlab-ingress-nginx-prod-controller
-
edit configs/packet-HA/fdo-gitlab-ingress-nginx.gotmpl
and re-enableloadBalancerIP:
-
helmfile -e packet-HA -l chart=ingress-nginx -i apply