`rtph264depay` element doesn’t handle multiple NALs in a FU as well as VLC and ffmpeg
I work for IPConfigure, a video management software company that heavily relies on GStreamer for streaming utilities.
We’ve recently come across an issue when using the rtph264depay
element while streaming H.264 from one of our camera vendors.
When streaming from this camera, no buffers containing I-Frames are pushed out of the rtph264depay element, resulting in corrupted output video. This was strange, as both FFmpeg and VLC display the video fine.
Digging further into what was going on, we discovered the camera fragments multiple NAL units over several RTP packets. Here is a screenshot of wireshark capture when streaming from this camera:
As seen above, there are multiple NAL units all within the same fragmentation unit, with the final NAL unit of this initial FU, the I-frame, continuing into the next 30 FUs. These NAL units are separated by Annex-B startcodes.
From RFC 6184 (RTP Payload format for H.264 Video), fragmentation is defined only for a single NAL unit, which means this camera is not abiding by the RFC.
The rtph264depay element is extracting the initial nal type of 7 (SPS) at the start of the FU, and storing that entire assembled FU as SPS data, when really that buffer contains multiple different NALs including an I-Frame.
Both VLC and FFmpeg support this stream despite the fact that it is non-RFC-compliant. Are there any preferred ways to get GStreamer to handle this situation similarly, and if not, would it be feasible for us to submit a patch that increases this element’s flexibility?