vaapidec: leak of dmabuf fds when transitioning from PLAYING to NULL state

Submitted by Sakari Kapanen

Link to original bug (#797152)

Description

Created attachment 373668
Reproduction of the described issue

When a pipeline containing a VAAPI decode element transitions from PLAYING to NULL state, it seems two dmabuf fds are leaked (when using EGL as the GL platform and dmabuf as the transfer mechanism from decode element to sink).

A minimal pipeline that reproduces the issue is the following:
videotestsrc ! x264enc ! video/x-h264 ! vaapih264dec ! video/x-raw(memory:DMABuf) ! fakesink

See the attached code which creates this pipeline and repeatedly sets it to PLAYING and NULL. Each time, the number of open dmabuf fds is printed. The output is as follows:

count of dmabuf fds: 0
setting pipeline state to PLAYING
count of dmabuf fds: 4
setting pipeline state to NULL
count of dmabuf fds: 2
setting pipeline state to PLAYING
count of dmabuf fds: 6
setting pipeline state to NULL
count of dmabuf fds: 4
setting pipeline state to PLAYING
count of dmabuf fds: 8
setting pipeline state to NULL
after exiting main loop
count of dmabuf fds: 6
after cleanup
count of dmabuf fds: 6

Each time the pipeline transitions to PLAYING, four dmabuf fds seem to be opened. When transitioning to NULL, only two are closed/freed. Additionally, when the program exits and the gstreamer elements are cleaned up (after three cycles), the fds still seem to be left open.

I have an application at hand where it is necessary to repeatedly stop and start the stream. This issue causes memory leakage and, eventually, program crashing due to too many open files. I have also tried to more gracefully stop the pipeline by sending an EOS event and only transitioning to NULL state after receiving the event on the pipeline's bus. However, the results are similar - leakage happens even then. Is there something regarding the dmabufs that has to be taken account during such transitions, or is this simply a bug in gstreamer-vaapi?

Even though the attached code uses a home-baked function that counts the open dmabuf fds, the behaviour can be verified by e.g. watching the output of lsof:
lsof -p $(pidof repro) | grep -c dmabuf

I would appreciate any input on how to resolve the issue (or if you can confirm it's an actual bug in gstreamer-vaapi or some other element).

At least the following elements exhibit the behaviour:
vaapih264dec
vaapih265dec
vaapivp8dec

Hardware:
Intel i5-8250U
Intel UHD Graphics 620

OS:
Arch Linux
gstreamer version 1.14.2
gstreamer-vaapi version 1.14.2

Attachment 373668, "Reproduction of the described issue":
repro.c

Version: 1.14.x