v4l2src + jpegdec dropping every second frame (timestamping / clocking / QoS issues?)
Submitted by Hector Martin
I'd been using GStreamer 1.12, which uses libv4l2, with this v4l2src usage:
gst-launch-1.0 v4l2src device=/dev/v4l/buffalo ! 'video/x-raw,width=1280,height=720,framerate=30/1' ! videoconvert ! fpsdisplaysink
AFAIK this works because the webcam outputs MJPEG and libv4l2 decodes it (the camera cannot output uncompressed video at 30FPS). GStreamer 1.14 disabled use of libv4l2 by default, though the above pipeline still works with GST_V4L2_USE_LIBV4L2=1. I used this with kind of basic pattern with a more complicated pipeline compositing multiple cameras and a screen capture.
However, trying to use gstreamer's own jpegdec instead, as is required without libv4l2, does not work properly (using libv4l2 with jpegdec doesn't work either). This is the test pipeline:
gst-launch-1.0 v4l2src device=/dev/v4l/buffalo ! 'image/jpeg,width=1280,height=720,framerate=30/1' ! jpegdec ! videoconvert ! fpsdisplaysink
I get 15 FPS output, and every second frame is dropped. This only happens with one of my two webcams - the other yields a clean 30 FPS. With GST_DEBUG=5 I see a line like this every second frame:
0:00:06.137449218 12416 0x55621e964590 DEBUG videodecoder gstvideodecoder.c:3155:gst_video_decoder_clip_and_push_buf:
<jpegdec0> Dropping frame due to QoS. start:0:00:04.839125944 deadline:0:00:04.839125944 earliest_time:0:00:04.854135807
Using "fpsdisplaysink sync=false" works around the issue and yields 30 FPS (for both cameras).
I get the feeling there's a clocking issue. Having both the webcam produce frames at a certain rate and the display consume them at its own rate (60Hz, which is a multiple of 30Hz) seems to make jpegdec think it's too late to the party after every output frame, on the next frame. Both webcams are nominally 30 FPS, but I guess one is probably slightly under and one slightly over that.
I'll attach debug logs for the second pipeline above, with both webcams (the one that causes dropped frames and the one that doesn't).