Nearly impossible to negotiate H.264 browser independently due to a flaw in webrtcbin
I already started several discussions w/o coming to a satisfying solution, e.g. here #1641 (closed)
The symptom is: If webrtcbin is supposed to answer to an incoming offer containing H.264 codecs it fails in almost all cases. No video is sent to remote. The other way around works.
The reason is, that the SDP comparison/negotiation is way too strict.
Problem 1
It is for instance not possible to have an SDP offer with payload type 106 w/o explicitly specifying this payload type while opening the pipeline, e.g. like so:
PIPELINE_TEST_H264 = '''
webrtcbin name=webrtcbin bundle-policy=max-bundle stun-server=stun://stun.l.google.com:19302
videotestsrc is-live=true pattern=smpte ! video/x-raw,width=640,height=480 ! videoconvert ! x264enc ! rtph264pay config-interval=1 !
application/x-rtp,media=video,encoding-name=H264,payload=106 ! webrtcbin.
If the payload=106 entry is missing, it will not work.
Problem 2
If the profile-level-id
of the OFFERER is not exactly a match with the profile-id returned after the local pipeline is opened it will not work.
For instance this trace shows, that the payload type 106 and profile-level-id 42c01e is returned after opening the a.m. pipeline
0:00:02.139386647 2346 0xb284bb20 DEBUG webrtcbin gstwebrtcbin.c:279:gst_webrtcbin_sink_event:<webrtcbin> On <webrtcbin:sink_0> checking negotiation? 1, caps application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, packetization-mode=(string)1, profile-level-id=(string)42c01e, sprop-parameter-sets=(string)"Z0LAHtkAoD2wFqDAwNSgAAADACAAAAeR4sXJ\,aMuMsg\=\=", payload=(int)106, ssrc=(uint)247990408, timestamp-offset=(uint)3232755765, seqnum-offset=(uint)10019, a-framerate=(string)30
Not only that the external H.264 codec needs to carry exactly the same 106 as payload type, I additionally need to munge the incoming SDP and change all constrained baseline profile-level-ids so that it matches my local profile-level-id, otherwise the video will not be sent:
Hack:
incoming_sdp = incoming_sdp.replace("42e01f", "42c01e") # test video
And this even though the 42c01e is a subset of the 42e01f, IMHO, so it MUST be accepted.
It is nearly impossible to have a solution with reasonable efforts to have this replacement done for each camera/source, since every input has it's own idea of a profile-level-id... I'm having 3 cameras and three profile-level-ids...
Problem 3
In addition it will not work if the OFFERER sends VP8 or VP9 codecs along side the H.264 codec definitions. This in turn requires to limit the offered codecs on the browser side to H.264 only, best just one profile.
The required functionality of setCodecPreferences
(https://developer.mozilla.org/en-US/docs/Web/API/RTCRtpTransceiver/setCodecPreferences) seems to be only experimental (even though supported by Chrome and Safari) and lacks at least getCapabilities
support in Firefox (https://developer.mozilla.org/en-US/docs/Web/API/RTCRtpSender/getCapabilities), so there is no support yet in all browsers for this kind of management.
All these reasons make it currently impossible to have an application using webrtcbin being able to ANSWER a valid incoming OFFER containing valid H.264 caps and agree on H.264 finally. As said: Working with an app just OFFERING H.264 works with all browsers.
Unless I have overseen something I would consider this as an issue.