Skip to content

webrtc: add raw payload support

François Laignel requested to merge fengalin/gst-plugins-rs:webrtc-raw-audio into main

Depends on

Only the last commit is specific to this MR, the other commits implement other features which serve as a basis for this work and come from the following MRs:

  • !1463 (merged): this is the base infrastructure for the precise example MR stack. It aims at providing the same features & convenience as the RTSP based rtp-rapid-sync-example for WebRTC. It helped improve the WebRTC C stack with support for intra CNAME synchronization & RFC 6051 in-band NTP-64 timestamps for rapid synchronization. The example also allows spawning an arbitrary number of audio and/or video streams.
  • !1500 (merged): adds RFC 7273 clock signalling to webrtcsink.

Raw payload support

This MR adds support for raw payloads such as L24 audio to webrtcsink & webrtcsrc.

Most changes take place within the Codec helper structure:

  • A Codec can now advertise a depayloader. This also ensures that a format not only can be decoded when necessary, but it can also be depayloaded in the first place.
  • It is possible to declare raw Codecs, meaning that their caps are compatible with a payloader and a depayloader without the need for an encoder and decoder.
  • Previous accessor has_decoder was renamed as can_be_received to account for codecs which can be handled by an available depayloader with or without the need for a decoder.
  • New codecs were added for the following formats:
    • L24, L16, L8 audio.
    • RAW video.

Open question

webrtcsink now proposes the raw codecs as part of its audio-caps & video-caps, after the encoder based codecs. E.g.:

     audio/x-opus
     audio/x-raw
                format: S24BE
                layout: interleaved
     audio/x-raw
                format: S16BE
                layout: interleaved
     audio/x-raw
                format: U8
                layout: interleave

The same was done for webrtcsrc with the audio-codecs & video-codecs. E.g.:

Default: "< (string)OPUS, (string)L24, (string)L16, (string)L8 >"

We could keep current defaults (e.g. 'audio/x-opus') and list the possible additional variants in the property's blurb, similarly to what webrtcsrc does:

audio-codecs: [...] Valid values: [OPUS, L24, L16, L8]

The webrtc-precise-sync examples were updated to demonstrate streaming of raw audio or video.

Edited by François Laignel

Merge request reports