baseparse/h264: basesink clipping one of the first NALUs because of DTS out of segment

Submitted by Marwen

Description

Created attachment 373871
The log for the first buffers

gst-launch-1.0 -v videotestsrc ! x264enc ! h264parse ! capsfilter caps="video/x-h264, stream-format=byte-stream, alignment=nal" ! fakesink silent=0

I enabled the logs for the parser and sink with GST_DEBUG=baseparse:7,h264parse:7,codecparsers_h264:7,basesink:7 to have the full picture. You can find the log for the first buffers attached.

The sink is dropping the second NALU of the stream because it fells out of the segment. That's because, in absence of a valid pts, the basesink is clipping on the dts which i don't think is a good idea since dts can fall out of the segment (especially for the first frames of h264 streams with b-frames).
The problem here comes also from the parser when transforming from a byte-stream format AU aligned to NAL aligned: when an AU is containing multiple NALU, only the first output buffer (NALU) is having the pts/dts of the input buffer(AU) while the following NALUs are having pts set to none and dts incremented by duration for every buffer. The attached log shows :

First input buffer to h264parse with size 7144:
--> baseparse gstbaseparse.c:2189:gst_base_parse_handle_buffer:<h264parse0> handling buffer of size 7144 with dts 999:59:59.933333334, pts 1000:00:00.000000000, duration 99:99:99.999999999
producing following buffers (NALUs):
--> baseparse gstbaseparse.c:2433:gst_base_parse_push_frame:<h264parse0> processing buffer of size 6 with dts 999:59:59.933333334, pts 1000:00:00.000000000, duration 0:00:00.033333333
--> baseparse gstbaseparse.c:2433:gst_base_parse_push_frame:<h264parse0> processing buffer of size 33 with dts 999:59:59.966666667, pts 99:99:99.999999999, duration 0:00:00.033333333
--> baseparse gstbaseparse.c:2433:gst_base_parse_push_frame:<h264parse0> processing buffer of size 9 with dts 1000:00:00.000000000, pts 99:99:99.999999999, duration 0:00:00.033333333
--> baseparse gstbaseparse.c:2433:gst_base_parse_push_frame:<h264parse0> processing buffer of size 761 with dts 1000:00:00.033333333, pts 99:99:99.999999999, duration 0:00:00.033333333
--> baseparse gstbaseparse.c:2433:gst_base_parse_push_frame:<h264parse0> processing buffer of size 6335 with dts 1000:00:00.066666666, pts 99:99:99.999999999, duration 0:00:00.033333333
Next input buffer to h264 with sie 5348:
--> baseparse gstbaseparse.c:2189:gst_base_parse_handle_buffer:<h264parse0> handling buffer of size 5348 with dts 999:59:59.966666667, pts 1000:00:00.133333333, duration 99:99:99.999999999

The main problem is that the second NALU (with buffer size 6) is clipped out in the basesink because it have no valid pts and its dts is falling out of the segment (start: 1000:00:00.000000000).
I'm not sure if it's better to set all NALUs to the same pts of the "parent" AU or set it to none, but the DTS is certainly set wrong.

So the questions are :

Should the basesink ever clip based on the DTS (even if no PTS is valid) ?
What should be the (PTS,DTS) of the NALUs coming from the same AU ?

Attachment 373871, "The log for the first buffers":
gst_log_h264.txt