v4l2: decoder can't release no display frame's timestamp or interlace frame
It was just the problem of timestamp/buffer tracking from the userspace.
- two fields or multiple slices into one CAPTURE buffer
multiple OUTPUT buffers generate one CAPTURE buffer: timestamp of the OUTPUT buffer queued first will be copied. It is not easy to track the decoding state of the second or future OUTPUT buffers.
- AV1, show existing frame An uncompressed frame header could do that. Return the corresponding OUTPUT buffer then dismiss its track would lead to a problem. Because it could be displayed later.
We would receive warnings from the code:
while ((oldest_frame = gst_video_decoder_get_oldest_frame (decoder)) &&
check_system_frame_number_too_old (frame->system_frame_number,
oldest_frame->system_frame_number)) {
if (oldest_frame->system_frame_number > 0) {
gst_video_decoder_drop_frame (decoder, oldest_frame);
oldest_frame = NULL;
if (!warned) {
g_warning ("%s: Too old frames, bug in decoder -- please file a bug",
GST_ELEMENT_NAME (decoder));
warned = TRUE;
}
More description: For VP9, AV1 we have golden frame, altref frame which would not be displayed but would be used as a reference frame. Unless we know we don't need to track their timestamp, it would be always in the decoder's frame queue. This more sounds like a problem of v4l2 itself, I think we need an event here to inform whether those non-display frames have been decoded successfully.
Besides, we would like to put the fields which belong to the same frame in a CAPTURE buffer interlaced. Following the v4l2 spec, we should use the timestamp of the first field, which leads to the GstVideoCodecFrame of the second field can't be popped.
Although working on v4l2videodec itself and the parser could fix the second problem, it would never be a good answer for the filed missing case.