SOLVED: Webrtcbin: Huge increasing memory leak while using appsink instead of autovideosink

I'm referring to the Java sample app: https://github.com/centricular/gstwebrtc-demos/tree/master/sendrecv/gst-java

With respect to increasing memory consumption everything is fine and balanced if we are using this approach to display the received video in a pop-up window:

if (name.startsWith("video")) {
            Element queue = ElementFactory.make("queue", "my-videoqueue");
            Element videoconvert = ElementFactory.make("videoconvert", "my-videoconvert");
            Element autovideosink = ElementFactory.make("osxvideosink", "my-autovideosink");
//            Element autovideosink = ElementFactory.make("autovideosink", "my-autovideosink");
            pipe.addMany(queue, videoconvert, autovideosink);
            queue.syncStateWithParent();
            videoconvert.syncStateWithParent();
            autovideosink.syncStateWithParent();
            pad.link(queue.getStaticPad("sink"));
            queue.link(videoconvert);
            videoconvert.link(autovideosink);
}

(autovideosink is replaced by osxvideosink for macOS)

If this sequence above is replaced by the following, the memory consumption of the Java app grows by 200 MB/s, which after a while leads to a complete crash of the app if not of the entire system:

if (name.startsWith("video")) {
            Element queue = ElementFactory.make("queue", "my-videoqueue");
            Element videoconvert = ElementFactory.make("videoconvert", "my-videoconvert");
            AppSink sink = (AppSink) ElementFactory.make("appsink", "my-appsink");
            sink.set("emit-signals", true);
            sink.connect(new AppSink.NEW_SAMPLE() {
                @Override
                public FlowReturn newSample(AppSink elem) {
                    Sample sample = elem.pullSample();
                    Structure capsStruct = sample.getCaps().getStructure(0);
                    String format = capsStruct.getString("format");
                    int width = capsStruct.getInteger("width");
                    int height = capsStruct.getInteger("height");

                    ByteBuffer bytes = sample.getBuffer().map(false);
                    byte[] buffer = new byte[bytes.remaining()];
                    bytes.get(buffer);
                    dragonfly.onNextFrame(format, buffer, width, height);

                    sample.dispose();
                    return FlowReturn.OK;
                }
            });

            sink.connect(new AppSink.NEW_PREROLL() {
                @Override
                public FlowReturn newPreroll(AppSink elem) {
                    Sample sample = elem.pullPreroll();
                    Structure capsStruct = sample.getCaps().getStructure(0);
                    String format = capsStruct.getString("format");

                    int width = capsStruct.getInteger("width");
                    int height = capsStruct.getInteger("height");

                    ByteBuffer bytes = sample.getBuffer().map(false);
                    byte[] buffer = new byte[bytes.remaining()];
                    bytes.get(buffer);
                    dragonfly.onNextFrame(format, buffer, width, height);

                    sample.dispose();
                    return FlowReturn.OK;
                }
            });

            pipe.addMany(queue, videoconvert, sink);
            queue.syncStateWithParent();

            videoconvert.syncStateWithParent();
            sink.syncStateWithParent();
            pad.link(queue.getStaticPad("sink"));
            queue.link(videoconvert);
            videoconvert.link(sink);
}

Is there anything wrong with the sequence above, which could cause the leak?

Sounds like a bug in the Java bindings, causing it to not free the buffers/samples correctly for whatever reason. Or maybe the GC is not run often enough so they all stay around for too long. Nothing wrong with the code above from what I can see, but copying every buffer instead of simply forwarding it is a quite wasteful activity.

In any case, this is not a support forum. Please use the mailing list for that https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel, or in this case even better the gstreamer-java communication forums.

closed

Why is this issue closed? I wasn't asking for support, I was reporting an issue, I'm seeing. And if the simple exchange of autovideosink by appsink causes a huge memory leak over time, and it all just happens with GStreamer, plugins-bad, webrtcbin - who should be involved either?

The copy is necessary, since the handling is done in native code and not under my control. And it is completely unrelated. The simple instantiation of the appsink causes the leak, even if we don't do anything.

Why is this issue closed? I wasn't asking for support, I was reporting an issue, I'm seeing. And if the simple exchange of autovideosink by appsink causes a huge memory leak over time, and it all just happens with GStreamer, plugins-bad, webrtcbin - who should be involved either?

The Java bindings. There's nothing wrong in the code above and the same works fine from C and other languages without memory leaks.

OK, so there is a special board for Java binding errors? Not sure if I understand. Who should be bothered with this problem, since I don't believe it is initially mine? For me it disqualifies the entire solution

To be more specific: This sequence is causing the leak:

if (name.startsWith("video")) {
            Element queue = ElementFactory.make("queue", "my-videoqueue");
            Element videoconvert = ElementFactory.make("videoconvert", "my-videoconvert");
            AppSink sink = (AppSink) ElementFactory.make("appsink", "my-appsink");

            sink.set("emit-signals", true);

            pipe.addMany(queue, videoconvert, sink);
            queue.syncStateWithParent();

            videoconvert.syncStateWithParent();
            sink.syncStateWithParent();
            pad.link(queue.getStaticPad("sink"));
            queue.link(videoconvert);
            videoconvert.link(sink);
}

Got this on the list you mentioned:

On Mon, 8 Jul 2019 at 10:57, Neil Young foreverneilyoung@googlemail.com wrote: Is there anything wrong with the sequence above, which could cause the leak?

You appear to be missing a call to Buffer::unmap before Sample::dispose?

You might want to use our mailing list at https://groups.google.com/forum/#!forum/gstreamer-java for Java specific questions. We're still somewhat semi-official! :-)

Best wishes,

Neil

But as as said: Even if no sample handling is involved, it leaks. Feeling like in a revolving door with GStreamer...

You have chosen to use third party bindings which are not maintained by the GStreamer project.

I don't know why you expect support for those by the GStreamer project.

If you can provide a test case for your problem in C or in any of the bindings we maintain such as python or Rust or c#, we'd be happy to have a look.

Who the hell should know about all these dependencies and links in this project? I'm just seen GStreamer at the first glance..

Anyway. I got the information, that the hint Neil gave on the other list fixes the link, so it seems, that the information I got regarding "the leak also happens if there is no handling at all" was wrong. I'm just the man in the middle here.

Closed

changed title from Webrtcbin: Huge increasing memory leak while using appsink instead of autovideosink to SOLVED: Webrtcbin: Huge increasing memory leak while using appsink instead of autovideosink

Who the hell should know about all these dependencies and links in this project? I'm just seen GStreamer at the first glance..

You need an overview of what software you're using in your application, especially in the first place you're using the GStreamer Java bindings and not GStreamer itself directly.

For future reference, the GStreamer Java bindings are maintained here: https://github.com/gstreamer-java

And its discussion forums are here: https://groups.google.com/forum/#!forum/gstreamer-java