Skip to content

RepoSet: Implement dedicated mechanism for mirroring

Mark Janes requested to merge majanes/mesa_ci:clone into master

Up to now, RepoSet has been re-used for three principal use cases:

  • checking out workspaces for build/test
  • mirroring repos in the build lab
  • polling branches to trigger builds

All of these use cases have serious bottlenecks and scalability problems, because a monolithic implementation cannot optimize git repositories for each use case.

This commit implements a mechanism for mirroring git repositories locally. For this use case, we have found that fetching all remotes into a single mirror is not scalable. Git performance declines as stale branches bloat the repo to +5GB.

Instead, create a separate bare mirror for each repo and remote. The complexity of the mirror is then limited by the size of the upstream repo.

The structure for the mirrors is:

{git_base_dir}/{project}/{remote_name}

For the origin repo, the remote_name is "origin"

Performance is dramatically improved by cloning mirrors in parallel threads.

Merge request reports