Large latency with webrtcbin
Hello, I'm trying to understand where the latency / failures come from with webrtcbin. It takes anything between 2-30 seconds to come up on local host on a web browser (firefox, linux). When it comes up, video starts to stream immediately, but sound lags for a few more seconds (sometimes up to 20), unless I use autoaudiosrc. audiotestsrc exhibits the same lag.
Code is basically just the webrtc h264 streaming example merged with the appsrc tutorial, with all the things I tried to make it not lag and that can be seen in the comments:
- pushing audio buffers explicitly instead of waiting for them to be pulled
- disabling a/v sync (at least I tried, by toggling the various sync / async fields on rtpbin)
- playing with the latency, frame-size, etc settings of opusenc, webrtcbin and friends
- playing with the audio / video priority
but nothing made any difference (at current HEAD of gstreamer as well as 1.20.2-1.1 as packaged by archlinux)
For simplicity here is the pipeline:
webrtcbin latency=10 name=webrtcbin stun-server=stun://stun.l.google.com:19302
videotestsrc is-live=1 pattern=snow
! videorate ! videoscale
! video/x-raw,width=64,height=64,framerate=60/1
! videoconvert
! queue max-size-buffers=1
! x264enc bitrate=600 speed-preset=ultrafast tune=zerolatency key-int-max=15
! video/x-h264,profile=constrained-baseline
! queue max-size-time=100
! h264parse
! rtph264pay config-interval=-1 name=payloader aggregate-mode=zero-latency
! application/x-rtp,media=video,encoding-name=H264,payload=96
! webrtcbin.
appsrc is-live=1 name=mysound
! audioconvert ! audioresample
! opusenc audio-type=restricted-lowdelay bandwidth=fullband bitrate=128000 frame-size=2.5
! rtpopuspay pt=97 ! webrtcbin.
Files:
Video of what I'm seeing: in the video, in the third try that finally loads at 00:35, sounds comes up quite a bit after the video (when the "Start feeding" messages appear in the log) - the HTML and element are synced on the same stream and both start outputting sound at the exact same time.