RFC: Media Source (MSE) API for GStreamer
Hello everyone,
I am working on a project that requires the use of the web MSE API [1], in native C/C++ code, outside the context of a web browser, in combination with a GStreamer playback pipeline.
We are considering adding this API on GStreamer, consisting of 2 parts:
- A library (in -bad) that implements the MSE API
- An element (webmsesource) that will act as the source element of the playback pipeline
The design and implementation of all of this is / will be based on Webkit's implementation of the same API.
[1]. https://www.w3.org/TR/media-source-2/
The library
The library is going to expose at least the following 4 public classes:
- GstMediaSource
- GstSourceBuffer
- GstSourceBufferList
- GstMediaSourceTrack
GstMediaSource is the top-level object that manages a list of GstSourceBuffer.
GstSourceBuffer accepts data that is provided by the application and internally launches a secondary pipeline to parse the provided data and fill in internal queues. It manages a list of GstMediaSourceTrack.
GstSourceBufferList expresses a list of source buffers and exposes signals that inform the application of buffers that were added or removed.
GstMediaSourceTrack is the equivalent of MediaSourceTrackGStreamer and TrackQueue from Webkit. It expresses a single parsed track, with a queue of parsed data that can be pulled from the playback pipeline. This is normally a private object in browsers, but we need to expose it here so that it can be linked to by the element that is going to act as the playback pipeline’s source.
The API of these classes is going to be very similar to the JavaScript ones in the MSE specification, but written in C and using the GObject / GStreamer programming conventions and style.
The source element
The source element is going to accept a GstMediaSource via a GObject property and refuse to operate without it. It will have to associate itself with the GstMediaSource (so that it can also be referenced from the MediaSource code) and will need to create streams (and post a GstStreamCollection) according to the available tracks on the MediaSource. For each stream, it will create a new src pad, and continuously pull data from the track's queue and push it in the pipeline.
(This is basically "webkitmediasrc" ported away from C++ and Webkit)
Comments? Questions?