aggregator, videoaggregator: force waiting even in live mode and other features

Submitted by Marianna S. Buschle

Description

In the recent Gstreamer conference (2016) I have presented a talk about using Gstreamer pipelines for image processing in industrial applications.
In this talk I mentioned that I had the need of muxing some metadata generated by processing (and thus modifying) image frames into the original (unmodified) images.

For this purpose I have created my own muxing element (using the Compositor as inspiration) which uses the VideoAggregator as the base class.
However the VideoAggreagtor and it's base class the Aggregator didn't quite cover my needs so I ended up modifying my own local copies of these 2 classes.

During the conference there was interest expressed in hearing which where my different use cases because maybe they could be added upstream.

Since I want to mux what is actually the same original frame (just divided and processed differently by 2 separate tee branches) I want to do timestamp matching of the buffers (I don't care about maintaining the fps and don't want to duplicate frames):

-Therefore I want to produce output frames only when I have received input frames in all pads (in contrast to the Aggregator producing output as soon as 1 pad has data, in case of a live source which is what I have).
My easy solution for this was adding a new property to the Aggregator that forces it to work as with non-live sources (so that it requires data in all pads).

-Then I added to VideoAggregator a property that defines if it should work "normally" (producing frames in a deadline, fps) or if it should check the buffer timestamps and only produce frames when they match. I have created a function like the fill_queues() function which does the timestamp checking.

-In case I'm only producing frames based on matched timestamps I have problems with QOS because the segment times might not match anymore (since frames are not necessarily produced regularly anymore). To solve this I have created a function that adjusts the segment times based on the produced frames.

-Lastly, in my "Compositor like" Muxer element I needed the option to drop frames in some cases so I have added a new GstFlowReturn to VideoAggregator (GST_FLOW_VIDEO_AGGREGATOR_DROP_FRAME == GST_FLOW_CUSTOM_SUCCESS) just like it exists for the BaseTransform

Version: 1.x