vaapipostproc + (brightness|saturation|contrast) + huge memory leak

Hi gstreamers,

i already wrote to the mailing list and received some useful hints for further diagnosis of a potential memory leakage. Now we're quite sure, that there is something wrong in vaapipostproc or vaapih264dec in combination with glupload and DMABuf when using color balancing of vaapipostproc in version 1.16.2.

Our pipeline looks like this:

auto pipeline = gst_pipeline_new( "pipeline" );

auto src = gst_element_factory_make( "udpsrc", "source" ); 
auto depay = gst_element_factory_make( "rtph264depay", "depay" ); 
auto dec = gst_element_factory_make( "vaapih264dec", "dec" ); 
auto postproc = gst_element_factory_make( "vaapipostproc", "postproc" ); 
auto upload = gst_element_factory_make( "glupload", "upload" ); 
auto sink = gst_element_factory_make( "appsink", "sink" );

gst_bin_add_many( GST_BIN(pipeline), src, depay, dec, postproc, upload, sink, NULL );

if( !gst_element_link_many( src, depay, dec, postproc, NULL ) ) { return -1; }

GstCaps* caps = gst_caps_from_string( "video/x-raw(memory:DMABuf), format=RGBA" );

if( !gst_element_link_filtered( postproc, upload, caps ) ) { return -1; }

if( !gst_element_link( upload, sink ) ) { return -1; }

// Setup Caps, Ports of src
// Inject GlContext
// Register Callbacks etc.

In case of a frequently change of vaapipostprocs properties, e.g. the brightness, contrast or saturation, we get a very high memory consumption, which leads to a kill by our watchdog. We reach dimensions about 2 GB, which are demanded by the gstreamer within a short period of 30s.

The color calibration is done by using the following property setter:

g_object_set( G_OBJECT( vaapipostproc ), "brightness", tmp, NULL );

To reproduce this issue it is merely nescessary to change one of the mentioned properties in a high frequent, periodic way.

for (int i = 0; i < 5000; i++)
{
    g_object_set( G_OBJECT( vaapipostproc ), "brightness", tmp, NULL ); 
    // Sleep is also possible
}

Even if a sleep is integrated in the loop, the memory isn't freed again. So a overload or a invalid use of calibration can be excluded.

Furthermore we tried several things to reproduce this issue in an isolated environment, which can produce reproducable result. We generated a small qt-project (with qmlsink), which is attached in this issue. demo_qmlsink_leakage.tar.xz We also tried to use trace logging for a leakage diagnosis but without any success. The log doesn't reveal anything helpful.

But in https://gitlab.freedesktop.org/gstreamer/gstreamer-vaapi/-/issues/106 a quite interesting test was provided, which demonstrated how DMABuf descriptors can be counted while the application is running. And yes, indeed - our FDs are increasing steadily when calibrating the color components.

We also tried integrate the following merge-requests:

merge_requests/251

merge_requests/276

merge_requests/303

Without any success, the error still occurs.

Help is very appreciated.

Best regards, Michael

demo_qmlsink_leakage.tar.xz