videodecoder: Don't push late frames

While it's a bit tricky to discard frames *before* decoding (because
we might not be sure which data is needed or not by the decoder), we
can discard them after decoding if they are too late anyway.

Any following basetransform based element or similar would drop the frame too.

gst-libs/gst/video/gstvideodecoder.c

@@ -3198,7 +3198,25 @@ gst_video_decoder_clip_and_push_buf (GstVideoDecoder * decoder, GstBuffer * buf)
     goto done;
   }
 
+  /* Is buffer too late (QoS) ? */
+  if (GST_CLOCK_TIME_IS_VALID (priv->earliest_time)
+      && GST_CLOCK_TIME_IS_VALID (cstart)) {
+    GstClockTime deadline =
+        gst_segment_to_running_time (segment, GST_FORMAT_TIME, cstart);
+    if (GST_CLOCK_TIME_IS_VALID (deadline) && deadline < priv->earliest_time) {
+      GST_DEBUG_OBJECT (decoder,
+          "Dropping frame due to QoS. start:%" GST_TIME_FORMAT " deadline:%"
+          GST_TIME_FORMAT " earliest_time:%" GST_TIME_FORMAT,
+          GST_TIME_ARGS (start), GST_TIME_ARGS (deadline),
+          GST_TIME_ARGS (priv->earliest_time));
+      gst_buffer_unref (buf);
+      priv->discont = TRUE;
+      goto done;
+    }
+  }
+
   /* Set DISCONT flag here ! */
+
   if (priv->discont) {
     GST_DEBUG_OBJECT (decoder, "Setting discont on output buffer");
     GST_BUFFER_FLAG_SET (buf, GST_BUFFER_FLAG_DISCONT);

[Grant Burkhart]

I can't say that I understand the algorithm used to discard buffers, but I can say that this change severely reduces the frame rate for the IMX gstreamer decoders. Perhaps there should be a parameter to toggle this behavior?

[Sebastian Dröge]

It simply drops frames here that would be considered too late in a video sink too when synchronizing to the clock. Your problem is probably a general synchronization problem in your pipeline somewhere.

Please create an issue with more details and a way to reproduce the problem.

[Grant Burkhart]

Thank you for your comments. I was just going to go with a version of code that doesn't have this change. But if I could track down a synchronization problem that would be much better. I have to admit I don't completely understand synchronization in gstreamer. I have submitted a bug: gstreamer/gst-plugins-base#668 (closed). I don't know how reproducible it will be without my specific hardware. Please let me know if I should add further information.