GitLab

The situation here is as follows:

1. RTMP stream that is streamed in real-time (but not live in GStreamer terms), 100ms interleave between audio and video
2. multiqueue configures 400ms interleave (1.5*100ms + 250ms) after the demuxere
3. avdec_h264 in frame threading mode uses one core per frame, leads to 700ms "interleave" after audio/video decoders

This has the following problems:

1. The pipeline gst-launch-1.0 uridecodebin3 uri=rtmp://... name=d use-buffering=true d. ! videoconvert ! fakesink silent=false name=video sync=true d. ! audioconvert ! fakesink silent=false name=audio sync=true -v usually does not pre-roll unless
   • A queue is added to the audio branch, or
   • interleave mode in multiqueue is disabled or the interleave is forced to be at least 1s, or
   • disable multi-threading in the decoder (max-threads=1), or using openh264dec for example
2. If downstream after the decoders is live (say, clocksync) then data will arrive too late once multiqueue has reconfigured the interleave (from default 5s to 400ms): previously video could go 700ms and more ahead into the decoder, now it will only go 400ms ahead which causes all further video frames to be delayed by 300ms (as audio and input advance in real-time)

I'm not sure what the solution here is. It's the interaction between decoders introducing considerable amounts of latency and multiqueue trying to be clever and estimating the interleave but ignoring everything after the demuxer.