Commit 600bf4b3 authored by Edward Hervey's avatar Edward Hervey 🤘
Browse files

migration work documents

* Switching the GStreamer project to gitlab
The GStreamer project uses a variety of services for development and
* The cgit repositories
* The bug tracker, where bug reporting and code
review happens.
* The Jenkins CI infrastructure, which tests on a
regular interval the GStreamer code.
* various other services (such as storage, websites,
mailing lists).
Unfortunately this had several downfalls:
* very poor integration between all the services and heavy burden on
the maintainers,
* "old" reviewing system,
* high entrance barrier for newcomers and patch contributors used to
more modern ways of developing/reviewing/contributing. Put simply
"people don't contribute because it's not as trivial as
* The jenkins CI system introduces too much developer overhead,
administrator maintenance, limiting proper "continuous" testing,
delivery, ...
In addition:
* GNOME is going to be turning off bugzilla (or rather making it
read-only) this year (2018), and is to be used
instead. While we could ask the GNOME administrators to keep the
GStreamer components in bugzilla writable for a bit longer, we can
not carry this for too long.
* has also setup a gitlab instance in order to ease
the burden of administration and provide a more integrated
Therefore we, as the GStreamer project maintainers and main
contributors, also intend to switch over most of our services to the GitLab setup.
The high priority items are the git repositories and the bug
tracker. The CI and websites are longer term efforts (we will carry
on using the existing infrastructure for those).
Our reasons for switching, and for switching to gitlab, also
coincides mostly with the reasons and are
also doing this migration. See these links for more information:
* Goals
** Make sure the development and maintenance workflow are clear
Go over all documents listing our current workflows:
* How people can contribute, report issues, etc..
* How developers (with git access) contribute and interact with
* How people do code review
* How releases are handled/done
Then go over all items and check:
* Can it be done in an identical fashion with gitlab ?
* If not, is there something close enough ?
* If not, propose a new way of dealing with it.
This should result in clear documentation.
*** existing documentation
[Contribution guide]
[Release doc]
** Backwards compatibility and legacy
While we do intend to switch to gitlab for many services, there are
a lot of data and information which will not be moved over but must
still be accessible for debugging/research/compatibility purposes.
Ensure that *all* bugzilla entries (including closed ones) are
* visible and accessible with the same URI
* searchable ?
gnome admins have stated they will switch to a read-only version.
cgit URI should remain accessible for all services, apps, scripts,
... that use the old URI (
freedesktop admins also want this, so it should be ok.
*** (long term)
Transform to a static page redirecting to the new gitlab CI
infrastructure. We don't really care too much about legacy data from
the old jenkins system.
*** (longer term)
"gitlab pages" offer a way to create website from git
repositories. which... is essentially what we do except that we use
custom scripts which need to be run manually by someone with
administrator access to freedekstop infrastructure.
Ideally we should switch to having CI scripts on gitlab.fdo that
automatically run those scripts if there are any changes and push
them to the proper gitlab "pages".
All links should end up being the same (i.e. we don't want to end up
with broken links on the website).
* Tasks
** Components to switch
The way gitlab works is by "projects", which would be under the
"GStreamer" group. Code, bug reports and merge requests are specific
to a project.
Go over the list of repositories and bugzilla "components" and
* Which repositories need to be ported. Some repositories can be
kept on the read-only cgit for legacy purposes but don't need to
be moved to gitlab.
* Which bugzilla components need to be ported and to which "project"
they correspond. Some components don't correspond to a git
repository for example. Maybe go over the list of bugs for those
components and either re-assign them to a valid component or
close them.
* Do we need a "administration" (or better naming) project which
would take care of high-level issues which aren't automatically
code-related (infrastructure comes to mind, or even the gitlab
transition itself) or cover feature requests that are accross the
whole project ?
This should result in a list of projects that are to be created on
*** List of git repositories
**** Repositories that won't be migrated
* attic/gst-android GStreamer-based replacement for stagefright (see cerbero for android build suppo...
* attic/gst-openmax GStreamer plug-in for OpenMAX IL API specification (deprecated, use gst-omx inst...
* attic/gst-plugins-gl GStreamer OpenGL libraries and plugins (obsolete, now part of gst-plugins-bad)
* attic/insanity GStreamer QA System (aka "Insanity") (obsolete, replaced by gst-validate in gst-...
* attic/insanity-gst Insanity GStreamer tests & helper functions (obsolete, replaced by gst-validate ...
* jhbuild GStreamer JHBuild module set
* sdk/cerbero Cerbero build system used to build the official upstream GStreamer 1.0 SDK binar...
* cerbero Cerbero build system used to build the official upstream GStreamer 1.0 SDK binar...
* common 'common' shared submodule
* gnonlin GStreamer Non-Linear plugins
* gst-build Build GStreamer and plugin modules using the Meson build system
* gst-ci Files and script for CI infrastructure
* gst-devtools Development and debugging tools for GStreamer
* gst-docs GStreamer documentation
* gst-editing-services GStreamer Editing Services Library
* gst-examples GStreamer example applications
* gst-ffmpeg GStreamer plugin for the libav* library (former FFmpeg)
* gst-integration-testsuites GStreamer Integration testsuites
* gst-libav GStreamer plugin for the libav* library (former FFmpeg)
* gst-omx OpenMAX IL GStreamer wrapper
* gst-plugins-bad 'Bad' GStreamer plugins and helper libraries
* gst-plugins-base 'Base' GStreamer plugins and helper libraries
* gst-plugins-good 'Good' GStreamer plugins
* gst-plugins-ugly 'Ugly' GStreamer plugins
* gst-python GStreamer Python binding overrides (complementing the bindings provided by pytho...
* gst-rtsp-server RTSP server based on GStreamer
* gst-streaming-server GStreamer Streaming Server
* gst-template Templates for applications and plugins
* gstreamer GStreamer open-source multimedia framework core library
* gstreamer-sharp C# bindings for GStreamer
* gstreamer-vaapi Hardware-accelerated video decoding, encoding and processing on Intel graphics t...
* orc Orc - Optimized Inner Loop Runtime Compiler
* qt-gstreamer Qt bindings for GStreamer
* www GStreamer website
*** List of bugzilla components
* cerbero
* common
* documentation
* don't know => gstreamer-project
* gnonlin
* gst-build
* gst-devtools
* gst-editing-services
* gst-editor
* gst-examples
* gst-libav
* gst-omx
* gst-plugins-bad
* gst-plugins-base
* gst-plugins-good
* gst-plugins-ugly
* gst-python
* gst-rtsp-server
* gst-sharp
* gstreamer (core)
* gstreamer-vaapi
* orc
* packages => cerbero
* qt-gstreamer
* www
* gst-android
* gst-monkeysaudio
* gst-plugins
* gst-plugins-gl
* gst-qa-system
* gst-rec
* gst-streaming-server
* gst-universe
* gstmm
** Bug conversion script
bztogl is a python script which is used by the gnome project to
transfer bugzilla bug reports into gitlab issues.
It is a good foundation, but needs to be adapted to:
* Use different gitlab hosts, since we don't want to upload to and would also need a different gitlab for
conversion testing
* Support bugzilla "components" , and specify to which gitlab
project a certain component would go.
We need to setup another gitlab setup (with outgoing mail disabled)
to test the conversion and get feedback before doing the final
** Workflow
Create a "gitlab" branch of gst-docs where documentation of the
whole workflow with gitlab will be detailed and discussed.
** Test project migration
* Pick a project (gst-docs ?) for which we attempt the conversion of:
* issues
* git repository
All of this in a test gitlab instance with the same users/privileges
** Notify all bugzilla users that they should create a valid account on gitlab ?
Or just notify by all standard means (IRC, mailing list, twitter,
...) possible with a safety margin (1/2 months) that they won't have
"proper" attribution when the conversion is run.
* Items to check in all open bugs
* priority : {'High', 'Low', 'Normal'} => Goes away
* keywords : {'STACKTRACE', 'newcomers', 'perf', 'probabledup', 'string', 'usability'}
* Labels to create : performance , newcomers/easytask ?
* see_also : List of URL, some are bugzilla entries
* severity : {'blocker', 'critical', 'enhancement', 'major', 'minor', 'normal'}
* target_milestone : The target milestone, can be converted to gitlab milestones
* New milestones for upcoming version => 1.XX.YY
* Or can we rename milestone and use 1.XX until the release and rename it to 1.XX.0 ?
* op_sys : {'All', 'BeOS', 'Linux', 'Mac OS', 'Windows', 'other'}
* status : {'ASSIGNED', 'NEEDINFO', 'NEW', 'REOPENED'}
* version : ( a ton of versions ... )
* Ignore, Add that information in the issue templates
* whiteboard :
Seems to be used by gstreamer-vaapi to set priorities ?
Used at some point also by some people, not sure we should keep it
=> They can create their own project labels in gstreamer-vaapi
* assigned_to : a few people
=> No problem. Check during script run which bugs have multiple assignees and change that
"[pluginname] something" => "pluginname: something"
* Issues/limitations
** Authors and dates
When doing the migration, when it comes to issues and notes
attributions we have to choose to have:
* Either proper attribution (if the user has an account) but wrong
date (the issue creation or the comment creation will have the same
date/time as when the conversion script was executed).
* *OR* proper dates but everything is attributed to a "Conversion
Script" account.
This is due to a limitation in the "sudo" feature of gitlab (which
allows you to behave on behalf of someone else). The only way to have
both would be to give admin powers to all accounts (on which we want
proper attribution and dates). I have little hope in getting this
done, because it would require essentially making the freedesktop
gitlab service temporarily offline (so that no *actual* user can get
admin privileges during the conversion).
*** Option to investigate : import/export
We should investigate if the gitlab project import/export feature
could be used.
This would allow creating on a private gitlab instance (with mailing
disable and access disabled):
* All the same user accounts as on gitlab.fdo
* Give all those accounts admin power (doesn't matter since they
won't be accessible)
* Run the bugzilla-to-gitlab import script (Which this time will
provide proper attribution *and* dates to all issues and comments).
* Remove admin power from all users (give them master access)
* Export that project from the private gitlab instance
* Import that project in the "public" gitlab instance (for testing
purposes it will not be gitlab.fdo, but eventually it will be).
** Fields and labels
Not all bugzilla fields have an equivalent gitlab "field", this list
goes over what does and doesn't map.
*** Compatible fields
* 'target_milestone' : This is covered by gitlab's "Milestone"
* 'status':
* ASSIGNED : There is an assignment property on issues
* NEW : Nothing specific, it's essentially not closed
* RESOLVED : Closed issue
* NEEDINFO : There isn't an existing field, would require a label
* 'assigned_to' : Use the assignment field
*** Incompatible fields
* 'see_also' : If it's a related bug entry, gitlab has a "related
issue" but it's only available in the "starter" Enterprise Edition.
The only option is to mark an entry in the different bugs
* 'priority' : DROP
* 'keywords' : Create well-known list of group-wide labels (see group labels)
* 'severity' : Create the appropriate labels
* Use labels from below if they exist, i.e. only blocker and enhancement (feature)
* 'op_sys' : If not generic or linux, we should have a few well-known
* Use labels from below
* 'version' : The version affected.
* Have users fill it in the report (Add to template)
*** Group labels used by (as reference)
* "1. Bug" Problems, incorrect behavior or appearance
* "1. Crash" The issue is a crash in the software
* "1. Feature" New features or capabilities to be added
* "1. Security" Security, privacy or safety issues
* "1. Blocker"
* "1. Regression"
* "1. Epic"
* "2. Needs Design" Needs an acceptable design solution to be identified
* "2. Needs information" Needs additional information to be diagnosed and/or resolved
* "2. RFC" Request For Comments and discussions from developers
* "3. Expected Behavior" Issue describes something that does not need to be changed
* "3. Incomplete" Necessary information is missing; issue cannot be confirmed/fixed
* #### "3. Not Gnome" Issue belongs to non-GNOME software
* "3. Out of Scope" Requested changes don't align with current technologies, designs or plans
* "4. Help Wanted" Help is welcome to resolve the issue
* "4. Newcomers" Tasks that are good for new contributors
* "5. <platform>" Android, ios, macOS, windows, linux, BSD
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment