Skip to content

GitLab

gstreamer
gstreamer
Open
Opened by Bugzilla Migration User@bugzilla-migration
Report abuse New issue

GstTocSetter: proposal to help standardize elements behaviour

Submitted by François Laignel @fengalin

Link to original bug (#791858)

Description

TOC Setter handling needs improvements in order to ease TOC related application developement and to standardize the behaviour of Elements that implement the GstTocSetter interface. This is a continuation of the discussion started in (0) and an attempt to trace and formalize what has been considered so far. This could also serve as guidelines for Elements that implement GstTocSetter.

Please correct me where I'm wrong and review my proposal for a simple enhancement. My idea is to extract parts of this description for the documentation of the solution.

Context

A Table Of Contents (1) is a tree structure which allows representing subdivisions of a media. Each TOC Entry is defined by its start and end positions, what it refers to (a chapter, an alternative angle, ...) and optional tags such as the title, the artist... An application can select a TOC Entry to reach the particular subdivision or aternative stream it refers to.

TOCs can be Global to a source or relative (Current) to the currently playing stream. An application can listen to the message bus of the Pipeline and get TOC messages (2).

From an Element's perspective, we can distinguish two main types for TOCs depending on their origin:

  • Upstream TOCs are received on sink pads via TOC Events (3).
  • User defined TOCs are assigned to the Element using the GstTocSetter interface (4). The User defined TOC should persist unaltered unless the user replaces it, resets it or if the Element is set back to NULL.

TOC Events also indicate an "updated" status which may be used by Elements to decide what to do with the received TOC.

Some media formats may not support all the features of the TOC model. It may be the case for alternative angles or the ability to define a Current TOC. An Element responsible for encoding to a format may need to adapt the upstream TOC to the format's capabilities. Moreover, there are multiple ways of representing a TOC structure. The Element should be flexible enough to map the input TOC's structure to fit its format's constraints.

Use Cases

The following use cases are identified for an Element which implements the GstTocSetter interface. We can consider a multiplexer such as the Matroska Multiplexer (muxer). This muxer is responsible for encoding a Matroska compliant output stream from multiple audio, video or subtitles source streams. Each source stream may send a TOC. The application requirements can be splitted into the following modes:

  • Use the Upstream TOC as the Output TOC. This is the usual behaviour when an application doesn't need to modify the TOC. Note the following variations which might require specific processing:

    • One Global Upstream TOC is received. The Element only needs to adapt the TOC to fit the format's capabilities.
    • One Current Upstream TOC is received. If the Element doesn't support this kind of TOC, it may need to merge it into a Global TOC.
    • Multiple Global / Current Upstream TOCs are received, possibly on different sink pads. How the Output TOC should be built is context specific. An Element may favor a particular sink pad, ignore subsequent TOC Events unless they are marked as "updated" or perform some sort of conflict resolution.

  • Use the User TOC as the Output TOC. In this mode, the Element must ignore Upstream TOCs and use the User TOC.

    • Just like for the Upstream TOC, the Element might implement specific processing to match the format's capabilities.
    • The application may need to use the Upstream TOC(s) to build the User TOC.

  • No Output TOC. The element must not output any TOC whatever the Upstream TOC(s) received or the User defined TOC.

Current Status

GstTocSetter allows storing a user TOC and resetting the stored TOC. Each GstTocSetter implementer handles the use cases in its own way. E.g.:

  • wavenc & flacenc use a dedicated field for the Upstream TOC. When the output stream is encoded, the User TOC is selected only if no Upstream TOC has been received. The user can't prevent an Upstream TOC from being encoded.
  • matroskamux stores the Upstream TOC in the GstTocSetter interface. In the proper state, the user can access the Upstream TOC, apply any modification or reset it. However, the User TOC can't persist unaltered.

As a workaround it is possible to define a sink pad probe (5) with GST_PAD_PROBE_TYPE_EVENT_DOWNSTREAM, then filter TOC events and return GST_PAD_PROBE_DROP to prevent the Upstream TOC from reaching the Element. The application can form a User TOC or do nothing if no TOC should be written. Another solution is to build a new TOC in the pad probe and replace the event with an event containing the new TOC.

Proposal

The proposal for this enhancement is to introduce a "selection_mode" field (and the associated getter and setter), a GstTocSelectionMode enum with the following possible values:

  • GST_TOC_SELECTION_UPSTREAM. This is the default when the GstTocSetter is created and the selection_mode will be set back to this upon reset. Elements should use the Upstream TOC if any and shouldn't encode any TOC otherwise.
  • GST_TOC_SELECTION_USER. The selection_mode switches to this value when a TOC is set on the TOC Setter. Elements should use the TOC Setter's TOC if any and shouldn't encode any TOC otherwise.
  • GST_TOC_SELECTION_NONE. In this mode, Elements shouldn't encode any TOC.

The workaround described in "Current Status" could be solution for applications which require to process the Upstream TOC(s) on the fly.

(0) https://bugzilla.gnome.org/show_bug.cgi?id=791736
(1) https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/GstToc.html
(2) https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/GstMessage.html#gst-message-new-toc
(3) https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/GstEvent.html#gst-event-new-toc
(4) https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/gstreamer-GstTocSetter.html
(5) https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/GstPad.html#gst-pad-add-probe