avfvideosrc: h264 support, optimize latency
Submitted by Joe Gorse
Link to original bug (#763011)
Description
I would like to add h264 output support to the avfvidesrc element (i.e. the AVFoundation Session within it) in order to improve the encode/decode latency. When using vtenc_h264 and vtdec_hw, the total pipeline latency is increased 200 ms (~6 frames). I am still new to how GStreamer manages buffers and all this, though in a direct application using native API's I'd expect 2-4 frames of additional latency for h.264 encoding and decoding for real-time streaming.
The following pipelines were used for these measurements:
With vtenc/vtdec and realtime settings
gst-launch-1.0 avfvideosrc device-index=0 ! "video/x-raw(memory:GLMemory),width=1280,height=720" ! gldownload qos=true ! vtenc_h264 realtime=true allow-frame-reordering=false ! vtdec_hw ! glimagesink
latency = 366 ms
Without vtenc/vtdec
gst-launch-1.0 avfvideosrc device-index=0 ! "video/x-raw(memory:GLMemory),width=1280,height=720" ! glimagesink
latency = 166 ms