Skip to content

Added W3C Media Source Extensions library

Jordan Yelloz requested to merge jyelloz/gstreamer:mse-merge-request-draft into main

This change adds a new library "mse" to gst-plugins-bad implementing the W3C Media Source Extensions specification. Applications can embed this library along with GStreamer in order to integrate software that uses the Media Source APIs without relying on a web browser engine. Typically an application consuming this library will wrap the C API with JavaScript bindings that match the Media Source API so their existing code can integrate with this library.

Closes #1312 (closed), the RFC behind this.

The design and implementation are based on WebKit's GStreamer implementation of this specification, converted to C, though the amount of layers relative to the WebKit implementation has been reduced.

High Level Design

The public APIs defined are Media Source, Source Buffer, and Source Buffer List which map directly to the concepts in the specification. The Media Source is the entry point to the API and maintains a group of Source Buffers. Applications feed raw data into the Source Buffers which then parse it as audio, video, and text streams that are stored in memory in their original encoding to be used by a media player.

The media player is anything that incorporates a GStreamer pipeline that has a msesrc source element inside. The msesrc is a special source element that can attach to a Media Source and consume the media data stored in its Source Buffers and forward the media through the pipeline.

Implementation Details

Each Source Buffer receives a stream of bytes from the application which are hopefully a valid multimedia stream. The Source Buffer uses an Append Pipeline to process the incoming data. The Append Pipeline is its own pipeline composed of an appsrc, a parsebin, and zero or more appsinks. As the stream is processed, the pipeline creates one appsink per track and forwards parsed samples back to the Source Buffer, which stores each track's samples in separate Track Buffers which organize each track's samples chronologically for future iteration. When the media player wants to play the stream, it will trigger tasks within each Source Buffer to feed samples from each requested track's Track Buffer into a Track Queue through a seek request to the Media Source. Each Track Queue is then polled by tasks within the msesrc and forwards samples down its pipeline for playback. The msesrc element is exported from the library in its own plugin and implements a uri handler on the mse:// protocol.

Status

In its current state the library is capable of handling MP4, WebM, and MP3 input data.

There are unit tests but these should be expanded to cover more possible scenarios. Also, an integration testsuite is needed as well. One goal is to assess this library's compatibility with the specification using the MSE compliance testsuite from the Web Platform Tests, though this will require some additional development due to the HTML/JavaScript dependency.

Not yet implemented

Media Source

  • Active Source Buffers property
  • Asynchronous signals

Source Buffer

  • Append Window modification
  • Timestamp offset support
  • [ ] Changing content type
  • Enforcement of content type constraints
  • Content eviction/removal
  • Asynchronous signals

gstmsesrc Element

  • Ready State tracking

Additional TODO items

  • Update copyright blocks to include WebKit authors
  • Add source-level documentation to public APIs
Edited by Olivier Crête

Merge request reports