Skip to content

`gst-libav`: Introduce infrastructure to allow `ffmpeg` codec parsers to be used in `gstreamer`.

Devin Anderson requested to merge surfacepatterns/gstreamer:ffmpeg-parsers into main

This commit makes it possible to easily create new parsers that use ffmpeg codec parsing functionality. The commit includes generic functionality that wraps ffmpeg parsers inside subclasses of GstBaseParse, which lowers the bar for creating ffmpeg based parsers.

There are a couple reasons for making this functionality available:

  1. gstreamer documentation states that:

    Parsers are demuxers with only one source pad. Also, they only cut the stream into buffers, they don't touch the data otherwise.

    https://gstreamer.freedesktop.org/documentation/plugin-development/element-types/one-to-n.html?gi-language=c

    ffmpeg has several "demuxers" that fit the gstreamer definition of a parser. In conformance with the rest of gstreamer, it makes more sense to make such functionality available as parsers instead of demuxers.

  2. Most ffmpeg decoders expect that the AVFrame instances they receive will each contain one (and only one) complete frame. There are exceptions - ffmpeg decoders can indicate that they can parse subframes (by adding AV_CODEC_CAP_SUBFRAMES to their capabilities field) - but very few ffmpeg decoders make use of this mechanism. If we want to make sure that the ffmpeg decoders made available in gstreamer function properly, then pipelines containing those decoders should also contains gstreamer elements that chop up data into buffers with single frames. As part of this effort, the source pad CAPS for the adapted parsers have a framed attribute set to TRUE.

(One of the implications of (2) is that, while the ffmpeg decoders included in gstreamer don't have a framed attribute set to TRUE in their sink CAPS, they almost certainly should.)

In addition to the parser infrastructure, this commit also introduces a G.723.1 parser so that the parsing infrastructure can be tested.

Upon reading the code, you'll notice that the G.723.1 parser is ranked as GST_RANK_MARGINAL. Like other adaptations of ffmpeg elements made available in gstreamer, the gstreamer adaptations of parsers are somewhat inefficient due to differing expectations between ffmpeg and gstreamer; in particular, before invoking the native ffmpeg parser, data is copied from a GstBuffer to a separate array that has extra padding that some ffmpeg parsers expect, and, once a frame is parsed, the frame is copied into a new GstBuffer that is then passed down the pipeline. Ranking these parsers as GST_RANK_MARGINAL allows future, more efficient parsers to be ranked higher than parsers made available through this parser infrastructure.

There's some history associated with this merge request. Last year, I tried to enable parsing of G.723.1 files by adding the ffmpeg demuxer for G.723.1 files, and fixing the CAPS for the decoder:

!3244

@tpm (quite correctly) pointed out several issues with enabling the demuxer in this way. When I had free time, I made a second attempt at adding G.723.1 support by making a merge request similar to this one:

!4315

Unfortunately, I made the mistake of emphasizing the G.723.1 support instead of emphasizing the parser functionality, which I believe led to quite a bit of confusion as to the nature of the merge request (apologies to @ndufresne). I hope that this merge request corrects the mistakes associated with the previous merge requests by emphasizing the parser functionality, and not including the G.723.1 decoder fixes (they'll be included in a separate merge request).

Finally, it'd be nice to use GstHarness for the tests included in this merge request, but it seems like the GstHarness API is push mode-centric. It wasn't clear to me how I'd easily write tests when pull mode was in use. Any advice on using the GstHarness when an element is in pull mode would be appreciated.

Merge request reports