Skip to content
GitLab
Closed
Issue created by Thibault Saunier@thiblahute🌵Maintainer14 of 14 checklist items completed14/14 checklist items

Merge all GStreamer modules into one single mono repository

GitLab didn't like having such a big merge request so I am going to open an issue instead of a MR, the branch described here can be found here:

This branch lays out the foundations for merging all our repositories into one.

This has been discussed in the community for a long time, and I think we have a consensus about the principle of a mono repo.

We should make the goal of this discussion to elaborate a practical, explicit consensus now. This is too important a decision to not have everyone in @gstreamer on board and aware of the implications.

Pros and cons of a monorepo

Pros:

  • Simplifies cross-module development, gst-build helps with that but there still exist friction points related to the current multi-repo approach
  • Simplifies CI a lot
  • Simplifies the build definitions a lot in the long run
  • Makes it clearer what the official GStreamer modules are
  • Makes it possible to have experimental internal APIs usable in all the gstreamer/ monorepo without exposing it outside GStreamer in the long run
  • Makes bisecting regressions much simpler
  • Avoids having to duplicate changes in each and every repo when some GLib symbols are deprecated for example, or in meson.
  • Simplifies the release process as we won't need to handle 20 different modules, but 1!

Cons:

  • Legal issues with ugly? How does FFmpeg handle these issues?

    We will still release the same modules so it shouldn't be a big problem

  • Some disruption in the development and packaging process, this is a one time thing though
  • The mono git repository is quite big (the .git is ~500MB)
  • No bisecting across the merge point

tim_entry_point()

Technical solution

To not lose any history and have all modules as one, without breaking single commits sha, merging repositories has been done with the following pseudo code (from the GStreamer core repository):

foreach GSTREAMER_MODULE
    git remote add GSTREAMER_MODULE.name GSTREAMER_MODULE.url
    git fetch GSTREAMER_MODULE.name
    git merge GSTREAMER_MODULE.name/master
    git mv list_all_files_from_merged_gstreamer_module() GSTREAMER_MODULE.shortname
    git commit -m "Moved all files from " + GSTREAMER_MODULE.name
endforeach

The exact code used can be found here, extended from @xclaesse

This approach has the following advantages:

  • checksum of each individual commit is preserved

  • we can cherry-pick commits from now merged modules, allowing pending merge requests to easily be reopened against the new monorepo

  • We can cherry-pick patches from the monorepo to the individual previous modules repos for stable releases doing:

    cd $GST_BUILD_PATH/gst-plugins-base
git remote add monorepo https://gitlab.freedesktop.org/gstreamer/gstreamer.git
git fetch monorepo
git cherry-pick <sha from monorepo>

Note: I am not using git subtree which does almost the same thing as what we manually do here (but in one single commit) because git log --follow some/file file doesn't work whereas it does with that solution. I can't think of a good reason to use it only that issue.

What does that branch implement

  • Merged modules with the new directory they landed in:

    • gstreamer -> subprojects/gstreamer
    • gst-plugins-base -> subprojects/gst-plugins-base/
    • gst-plugins-good -> subprojects/gst-plugins-good/
    • gst-plugins-bad -> subprojects/gst-plugins-bad/
    • gst-plugins-ugly -> subprojects/gst-plugins-ugly/
    • gstreamer-vaapi -> subprojects/gstreamer-vaapi/
    • gst-omx -> subprojects/gst-omx/
    • gst-libav -> subprojects/gst-libav/
    • gst-ci -> ci/
    • gst-rtsp-server -> subprojects/gst-rtsp-server/
    • gst-editing-services -> subprojects/gst-editing-services/
    • gst-devtools -> subprojects/gst-devtools/
    • gst-docs -> subprojects/gst-docs/
    • gst-examples -> subprojects/gst-examples/
    • gst-integration-testsuites -> subprojects/gst-tests/integration/
    • gst-python -> subprojects/gst-python/
    • gstreamer-sharp -> subprojects/gstreamer-sharp/
    • gstreamer-rs -> subprojects/gstreamer-rs/
    • gst-plugins-rs -> subprojects/gst-plugins-rs
    • gst-build -> scripts/ and tests/

  • The gstreamer core module is very close to what gst-build was in term of implementation

  • cerbero has been ported in cerbero!653 (closed)

  • CI has been ported over

  • Build products parity

What is known to be missing

  • Moving more files around manually: remove some READMES, add toplevel LICENSE/ etc..
  • Fix documentation (although that was started already)
  • See how we can better handle modulename=auto in meson
  • Mass move issues from the different modules to the gstreamer project (should be simple to do in the UI)

Plan

  • Agree on the wanted structure of the mono repo
  • Generate the monorepo again with latest GStreamer
    • Update to latest CI and make sure it passes - last 'scheduled' pipeline can be found here
      • includes everything from latest gst-ci
      • includes everything from latest gstreamer-rs (only executed on 'scheduled' jobs
      • details about cerbero related job can be found here
    • Build a cerbero branch
    • Update the documentation
  • Do mock releases of the various modules (Issue in meson to make it work fixed in https://github.com/mesonbuild/meson/pull/7907) (do later)
  • Send a mail to the gstreamer-devel mailing list to let the broader community about the plan This needs to contain a summary of the plan - Thread available here
  • Mass move issues from the different modules to the gstreamer project (simple to do in the UI) (No automatic mass-move of issues, we want to triage them, see monorepo FAQ)
    • We should add tags to the issues before moving them (Tags have been added)
  • Write a message on all opened MRs explaining how to move them to the mono repo and linking to this issue (No automatic mass-comment on open MRs, let's have active contributors move MRs via the script first, collect feedback to improve the script and then over time triage the remaining MRs and suggest they be moved and/or post link to script at a later time)
    • First try to make a script to move open MRs into GStreamer core
  • Get https://github.com/mesonbuild/meson/pull/7907 merged
  • Create a temporary monorepo branch on origin (as proposed here)
    • Open MR against that monorepo branch for review
  • Once the monorepo is merged we should update READMEs in all merged repos to link with gstreamer core (clean-up for later; N.B. not all READMEs are the same)

Misc notes

  • For modules that do not follow the exact same release cycle (only gstreamer-rs) the releases tag should be prefixed. EDIT - We won't make gstreamer-rs part of the monorepo for now as decided discussed here
Edited by Tim-Philipp Müller