Rtpjitterbuffer memory leak
I recently took over a project that uses GStreamer to record security cameras. The previous developer was having some issues with a certain camera failing after approximately 13 hours of recording. I believe some of the issues he saw were explained here: Memory Leak?. I don't really have a lot of knowledge about the particular problem, but he did leave me enough to reproduce the problem.
GStreamer versions tested (both have the same issue):
- Pre-built Windows 1.14.3
- Built from source Windows 1.19.3
Operating system is Windows 10 Pro, build 19042.
We have multiple cameras of different models and brands. The following pipeline works perfectly fine on a Hikvision DS-2DE3304W-DE, firmware V5.5.71 build 180725:
gst-launch-1.0.exe --gst-debug-level=2 rtspsrc location=rtsp://hikvision_camera_address:554 name=src ! rtpjitterbuffer ! rtph264depay ! queue ! splitmuxsink muxer=mpegtsmux sink=filesink location="hikvision%04d.mp4" max-size-bytes=5242880 max-files=4 name=mux
The camera with an issue is a Ubiquiti UVC, firmware UVC.v3.5.2.63.5d0239d.161121.1655. The failing pipeline is:
gst-launch-1.0.exe --gst-debug-level=6 rtspsrc location=rtsp://ubiquiti_camera_address:554/s0 name=src ! rtpjitterbuffer ! rtph264depay ! tee name=vtee ! queue ! fakesink vtee. ! queue ! splitmuxsink mux-overhead=0 muxer=mpegtsmux sink=filesink location="ubiquiti%04d.mp4" max-size-bytes=5242880 max-files=4 name=mux src. ! rtpjitterbuffer ! rtpmp4gdepay ! aacparse ! tee name=atee ! queue ! fakesink atee. ! queue ! mux.audio_1
When running with debug level 2, it seemed like the problem was a "rtpjitterbuffer gmem.c failed to allocate" message. There was some other message logged at the same time about not being able to allocate a certain number of bytes of memory. I'd have to re-run the pipeline to get the exact message. It wasn't much memory. Something like a megabyte I believe. Unfortunately I wasn't near my computer when the memory allocation error happened so I don't know if process memory was shooting upwards. However, I had checked it a few hours into the recording and memory was staying constant, as well as handle count.
I'm suspicious of the Ubiquiti camera since the Hikvision doesn't have a problem and the pipelines don't strike me as that much different other than the Ubiquiti has audio recording. However, even if the Ubiquiti camera is putting out some garbage data, I don't think GStreamer should be failing with a memory error.
HERE is a link to a zip file of stdout and stderr from the above debug level 6 pipeline. It's admittedly a very large error file. I'm hoping someone can spot an issue. I haven't been able to find where in the file the memory error happened. It doesn't appear to be the same output as with debug level 2. But the problem is repeatable so I think something has to be there. There are some CRITICAL level events, but I'm not sure how they relate to the memory issue.
I've asked my boss if there's any way to provide access to this camera for testing, but I haven't heard back from him. However, I am willing to try any suggested code changes in my 1.19.3 build to see if they will solve the issue.