gstwebrtcbin.c: retransmissions work with bundle-policy=max-bundle but not with default bundle-policy
Hello,
I've found a bug in gstwebrtcbin.c (https://github.com/GStreamer/gst-plugins-bad/blob/master/ext/webrtc/gstwebrtcbin.c) which causes retransmissions (RTX for NACKs) to not work in common scenarios. I've also found the root cause and provide a detailed explanation:
Setup:
- Gstreamer installation based on restreamio/gstreamer:1.19.2.0-prod Docker image
- do-nack is set to true at the transceiver (fec-type is not set as I'm testing retransmissions only)
- gst1-java-core:1.4.0 is used to build up the pipeline
- Gstreamer is used to send a video stream (video only, no audio, send-only) to Chrome
- A range of multiple video source is used (rtspsrc, srtsrc, videotestsrc, ...)
- Chrome on Windows is used for receiving a video-stream via WebRTC
- H264 encoding (baseline and high profile) is used in different cases
- Clumsy is used for simulating packet loss (client-side)
SDPs:
The SDPs are correctly mentioning nack and rtx and pt mappings seem to be correct.
SDP Offer:
v=0
o=- 8338404406669528263 0 IN IP4 0.0.0.0
s=-
t=0 0
a=ice-options:trickle
m=video 9 UDP/TLS/RTP/SAVPF 97 96
c=IN IP4 0.0.0.0
a=setup:actpass
a=ice-ufrag:qt8YBq9p6brgGFJPTcOEZnpzNuGGaiZc
a=ice-pwd:IUzbx6zU7c1cAEpr9v8pUd8fQX4WMWc4
a=rtcp-mux
a=rtcp-rsize
a=sendonly
a=rtpmap:97 H264/90000
a=rtcp-fb:97 nack
a=rtcp-fb:97 nack pli
a=rtcp-fb:97 transport-cc
a=framerate:25
a=fmtp:97 packetization-mode=1;profile-level-id=4d001f;sprop-parameter-sets=Z00AH5Y1QKALdNwEBAUAAAcIAAFfkAQ=,aO48gA==
a=rtpmap:96 rtx/90000
a=fmtp:96 apt=97
a=ssrc-group:FID 2 876927395
a=ssrc:2 msid:user73081093@host-10bea89f webrtctransceiver4
a=ssrc:2 cname:user73081093@host-10bea89f
a=ssrc:876927395 msid:user73081093@host-10bea89f webrtctransceiver4
a=ssrc:876927395 cname:user73081093@host-10bea89f
a=mid:video0
a=fingerprint:sha-256 5A:BB:64:EB:DA:4C:59:AE:25:97:46:BA:18:BC:67:A1:8A:8B:30:15:B0:D3:F0:21:7C:43:95:79:DA:EA:66:25
a=rtcp-mux-only
SDP Answer:
v=0
o=- 9170499300271825649 2 IN IP4 127.0.0.1
s=-
t=0 0
a=msid-semantic: WMS
m=video 9 UDP/TLS/RTP/SAVPF 97 96
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:eFrN
a=ice-pwd:8X4CVq0UQPKCyHgT9Mt+ArfF
a=ice-options:trickle
a=fingerprint:sha-256 9E:43:31:2F:55:64:63:76:53:B8:35:F9:9B:B9:AD:71:04:95:E2:F9:2A:66:71:2D:64:B0:26:42:E8:4D:CD:73
a=setup:active
a=mid:video0
a=recvonly
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:97 H264/90000
a=rtcp-fb:97 transport-cc
a=rtcp-fb:97 nack
a=rtcp-fb:97 nack pli
a=fmtp:97 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=4d001f
a=rtpmap:96 rtx/90000
a=fmtp:96 apt=97
I can see that NACKs are sent by Chrome via chrome://webrtc-internals
The NACK count is increasing when packets are lost.
So far so good, but the retransmissions through Gstreamer don't work yet.
I've debugged it and finally found the reason in gstwebrtcbin.c:
on_rtpbin_request_aux_sender calls _set_rtx_ptmap_from_stream (webrtc, stream) after creating the rtprtxsend (rtx = gst_element_factory_make ("rtprtxsend", NULL)) while stream->rtxsend has not yet been set.
It's basically set at the end of on_rtpbin_request_aux_sender: stream->rtxsend = gst_object_ref (rtx);
I can verify that _set_rtx_ptmap_from_stream is called only once while rtprtxsend is null via the logs (GST_DEBUG='3,rtprtx*:5,webrtc*:6'):
0:01:19.678286676 6 0x7fd1c42d7060 DEBUG webrtcbin gstwebrtcbin.c:4313:_set_rtx_ptmap_from_stream:<transportstream0> setting payload map on (NULL) : (NULL) and application/x-rtp-pt-map, 97=(uint)96;
The following logs show that the retransmissions triggered by NACKs are not working:
0:01:19.679474693 6 0x7fd1c53bc700 WARN rtprtxsend gstrtprtxsend.c:655:gst_rtp_rtx_send_sink_event:<rtprtxsend0> Payload 97 not in rtx-pt-map
and
0:08:28.606424318 6 0x7fd1a1176c60 DEBUG rtprtxsend gstrtprtxsend.c:489:gst_rtp_rtx_send_src_event:<rtprtxsend0> got rtx request for seqnum: 31149, ssrc: 2
0:08:28.606434319 6 0x7fd1a1176c60 WARN rtprtxsend gstrtprtxsend.c:525:gst_rtp_rtx_send_src_event:<rtprtxsend0> requested seqnum 31149 has not been transmitted yet in the original stream; eithe
r the remote end is not configured correctly, or the source is too slow
0:08:28.606442719 6 0x7fd1a1176c60 DEBUG rtprtxsend gstrtprtxsend.c:489:gst_rtp_rtx_send_src_event:<rtprtxsend0> got rtx request for seqnum: 31150, ssrc: 2
0:08:28.606447819 6 0x7fd1a1176c60 WARN rtprtxsend gstrtprtxsend.c:525:gst_rtp_rtx_send_src_event:<rtprtxsend0> requested seqnum 31150 has not been transmitted yet in the original stream; eithe
r the remote end is not configured correctly, or the source is too slow
The lost packets are not received by Chrome too and the video stream starts freezing already at <5% packet loss.
This is problematic because _set_rtx_ptmap_from_stream only sets the "playload-type-map" if stream->rtxsend is set which is not the case:
if (stream->rtxreceive)
g_object_set (stream->rtxreceive, "payload-type-map", pt_map, NULL);
if (stream->rtxsend)
g_object_set (stream->rtxsend, "payload-type-map", pt_map, NULL);
A workaround is to use bundle-policy=max-bundle because in this case _set_rtx_ptmap_from_stream is called a second time from _update_transceiver_from_sdp_media:
if (!bundled || bundle_idx == media_idx) {
if (stream->rtxsend || stream->rtxreceive) {
_set_rtx_ptmap_from_stream (webrtc, stream);
}
g_object_set (stream, "dtls-client",
new_setup == GST_WEBRTC_DTLS_SETUP_ACTIVE, NULL);
}
The workaround can be verified via logs too:
The "payload-type-map" is set via _set_rtx_ptmap_from_stream at the second call (from _update_transceiver_from_sdp_media) with stream->rtxsend
being not null:
0:01:11.110764316 7 0x7f9220b115e0 DEBUG webrtcbin gstwebrtcbin.c:4313:_set_rtx_ptmap_from_stream:<transportstream0> setting payload map on (NULL) : (NULL) and application/x-rtp-pt-map, 97=(uin
t)96;
0:01:11.111793531 7 0x7f9220b115e0 DEBUG webrtcbin gstwebrtcbin.c:4313:_set_rtx_ptmap_from_stream:<transportstream0> setting payload map on (NULL) : <rtprtxsend0> and application/x-rtp-pt-map,
97=(uint)96;
The requested packets are found and retransmitted:
0:02:35.661652056 7 0x7f91fc840d20 DEBUG rtprtxsend gstrtprtxsend.c:489:gst_rtp_rtx_send_src_event:<rtprtxsend0> got rtx request for seqnum: 45671, ssrc: 2
0:02:35.661658856 7 0x7f91fc840d20 DEBUG rtprtxsend gstrtprtxsend.c:405:gst_rtp_rtx_buffer_new:<rtprtxsend0> creating rtx buffer, orig seqnum: 45671, rtx seqnum: 58534, rtx ssrc: 851211DE
0:02:35.661680057 7 0x7f91fc840d20 DEBUG rtprtxsend gstrtprtxsend.c:489:gst_rtp_rtx_send_src_event:<rtprtxsend0> got rtx request for seqnum: 45672, ssrc: 2
0:02:35.661693457 7 0x7f91fc840d20 DEBUG rtprtxsend gstrtprtxsend.c:405:gst_rtp_rtx_buffer_new:<rtprtxsend0> creating rtx buffer, orig seqnum: 45672, rtx seqnum: 58535, rtx ssrc: 851211DE
0:02:35.661705557 7 0x7f91fc840d20 DEBUG rtprtxsend gstrtprtxsend.c:489:gst_rtp_rtx_send_src_event:<rtprtxsend0> got rtx request for seqnum: 45673, ssrc: 2
0:02:35.661715557 7 0x7f91fc840d20 DEBUG rtprtxsend gstrtprtxsend.c:405:gst_rtp_rtx_buffer_new:<rtprtxsend0> creating rtx buffer, orig seqnum: 45673, rtx seqnum: 58536, rtx ssrc: 851211DE
0:02:35.661726757 7 0x7f91fc840d20 DEBUG rtprtxsend gstrtprtxsend.c:489:gst_rtp_rtx_send_src_event:<rtprtxsend0> got rtx request for seqnum: 45674, ssrc: 2
Now the retransmission is working and the video stream is still stable at >20% packet loss.
I would appreciate if someone from the Gstreamer community could verify my bug report and fix it.
Thank you!
Best regards,
Markus Pollak