Slow but steady memory increase when running pipeline in Python
Describe your issue
My team has been trying to understand the source of a slow but steady increase in resident set size of our application when running a gstreamer pipeline that reads from an RTSP source and writes out 10 seconds MPEG-TS segments to disk via Python. The program consumes about 10MB of additional resident set size per hour.
Expected Behavior
Steady resident set size when running our pipeline over the course of many hours of continuous usage.
Observed Behavior
Steadily increasing resident set size of ~10MB/hour.
Setup
- Operating System: Ubuntu 22.04 (Linux b2a31bb40923 5.15.49-linuxkit #1 SMP PREEMPT Tue Sep 13 07:51:32 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux)
- Device: Computer
- GStreamer Version: 1.20.3
- Command line: N/A
Steps to reproduce the bug
We decided to pare our application down to a minimal implementation that basically just creates a Gst.Pipeline and runs it. This minimal implementation still results in the same resident set size increase. The process starts at around 35MB of resident set and steadily increases to about 55MB after two hours of continuous runtime.
Here's the code, if you're curious:
import gi
gi.require_version("Gst", "1.0")
from gi.repository import GLib, Gst
def main() -> None:
Gst.init(None)
pipeline_str = (
"rtspsrc location=rtsp://admin:supersecretpassword@192.168.10.250/media/video2 protocols=tcp "
"! rtph264depay wait-for-keyframe=true request-keyframe=true "
"! h264parse "
"! splitmuxsink name=splitmuxsink max-size-time=10000000000 send-keyframe-requests=true muxer=mpegtsmux location=/data/test%05d.ts"
)
pipeline = Gst.parse_launch(pipeline_str)
loop = GLib.MainLoop()
pipeline.set_state(Gst.State.PLAYING)
print("Started pipeline")
try:
loop.run()
except Exception:
pass
pipeline.set_state(Gst.State.NULL)
print("Stopped pipeline")
if __name__ == "__main__":
main()
How reproducible is the bug?
Always
Screenshots if relevant
Solutions you have tried
Interestingly, running the same pipeline under gst-launch-1.0
does not seem to result in an increase in resident set size. After 16 hours of execution, the resident set size of gst-launch-1.0
stayed at about 21MB.
In desperation, we tried removing splitmuxsink
and replacing it with fakesink
. To our surprise, the increase in resident set size went away!
So it seems like there's something about running splitmuxsink
under Python that is causing an issue. Running the Python process with GST_TRACERS=leaks
and GST_DEBUG=GST_TRACER:7
did not show any leaked gstreamer memory. We've tried using tracemalloc
and memray
to inspect the Python memory allocation before and after running the pipeline for a few hours, but it basically shows nothing of interest, although admittedly we may not be using these tools correctly.