rtpjitterbuffer: RTCP SRs with "out-of-order" RTP timestamps generate bad reference timestamps
Describe your issue
Duplicate reference timestamp metadata is sometimes seen on consecutive frames in my GStreamer RTSP video client running a pipeline with the following front-end elements: rtspsrc add-reference-timestamp-meta=true ! rtph264depay ! h264parse ! ...
Expected Behavior
Unique, correct reference timestamp metadata for all frame buffers coming out of the h264parse src pad.
Observed Behavior
Multiple instances of duplicated, incorrect reference timestamp metadata coinciding with RTCP SRs received by the video client.
Data from deeper investigation
Some RTSP servers (some Axis cameras, GStreamer RTSP server under certain conditions) can generate RTCP Sender Reports whose RTP timestamps are not strictly ordered in sequence with the accompanying RTP stream. When this happens, the gstrtpjitterbuffer code will generate bad reference timestamp metadata for some number of packets until the RTP stream timestamps have "caught up" to the last RTCP SR timestamp. For example, take the following packets received in order:
RTP packet N: Time=100000
RTP packet N+1: Time=100000
RTP packet N+2: Time=100000
RTCP SR packet: RTP timestamp=115500 NTP time=<foo>
RTP packet N+3: Time=100000
RTP packet N+4: Time=103000
RTP packet N+5: Time=103000
RTP packet N+6: Time=106000
...
RTP packet N+M: Time=115000
RTP packet N+M+1: Time=118000
In this case, all of the buffers for packets from N+3 to N+M will have identical improperly-calculated reference timestamp metadata due to the following code in gstrtpjitterbuffer.c:
// If we can calculate a NTP time based solely on the Sender Report, or
// inband NTP header extension do that so that we can still add a reference
// timestamp meta to the buffer
if (!GST_CLOCK_TIME_IS_VALID (ntp_time) &&
GST_CLOCK_TIME_IS_VALID (priv->last_known_ntpnstime) &&
priv->last_known_ext_rtptime != -1) {
guint64 ext_time = priv->last_known_ext_rtptime;
ext_time = gst_rtp_buffer_ext_timestamp (&ext_time, rtptime);
ntp_time =
priv->last_known_ntpnstime + gst_util_uint64_scale (ext_time -
priv->last_known_ext_rtptime, GST_SECOND, priv->clock_rate);
}
When the SR RTP timestamp is later than the RTP buffer timestamp, this code cannot calculate the correct NTP reference timestamp correctly because it subtracts the RTCP SR RTP timestamp from the RTP buffer timestamp, returning the result as a scaled unsigned 64-bit int. When the SR RTP timestamp is greater than the RTP buffer timestamp, the gst_util_uint64_scale() function returns 0. Until the SR RTP timestamp is less than the RTP buffer timestamps, all buffers will have reference timestamp metadata incorrectly set to the priv->last_known_ntpnstime (from the RTCP SR).
Setup
- Ubuntu 22.04
- RTSP client: laptop
- RTSP server: Axis M-3068P video camera (also seen with GStreamer RTSP server serving video generated from testvideosrc with RTSP server rtpbin property "rtcp-sync-send-time" set to false)
- GStreamer git b90d0274
- Running our own application, but likely reproducible with a gst-launch pipeline of: rtspsrc location="" latency=40 add-reference-timestamp-meta=true buffer-mode=none ntp-sync=true protocols=tcp ! rtph264depay ! h264parse disable-passthrough=true ! video/x-h264,stream-format=byte-stream,alignment=au ! fakesink
Steps to reproduce the bug
Connect a GStreamer rtspsrc pipeline to an RTSP server that generates RTCP SRs with RTP timestamps that are "out-of-order" with respect to the associated RTP stream.
How reproducible is the bug?
Always reproducible with Axis M3068-P camera and GStreamer git. Sometimes reproducible with GStreamer RTSP server.
Solutions you have tried
I have actually implemented a tentative fix in gstrtpjitterbuffer.c that will delay updating the priv->last_known_ntpnstime and priv->last_known_ext_rtptime until the RTP buffer timestamps are greater than the RTCP SR timestamp. I will attach a patch of this tentative fix for consideration.