Usage statistics for gitlab.fd.o
We have a massive growth in usage, and we need to figure out whom to blame for this so that optimizatio efforts are actually directed where we'd benefit most, instead of just wild guesses.
First priority is google cloud storage downloads (since that's growing the quickest). We'd want to be able to assign used network bw both to projects (to know where to cut artifact sizes) and to external CI runner labs (in order to know where we really need better caching).
2nd priority would be figuring out where the GCE egress bw goes towards. See also #8 Note that @daniels says that raw logs are already available if someone wants them.
Todo
-
start collecting google cloud storage logs (@daniels took care already) -
improve grafana scripts for cloud storage to split out direct downloads vs GCE traffic -
upgrade nginx setup so we have useful logs, see #50 (closed) -
start collecting logs for GCE network egress -
provide them somwhere (we have http://fdo-grafana2.banquise.eu/ should this be part of fd.o infra proper?)-> https://grafana.freedesktop.org -
find volunteer to crunch through them, probably want to share the scripts somewhere too on a freedesktop repo. -> https://gitlab.freedesktop.org/freedesktop/fdo-logs-analysis