pipewiresrc: internal queue is growing indefinitely
Created by: tretter
The pipewiresrc element contains an internal queue for buffers that have been processed by Pipewire but have not been retrieved by GStreamer. This queue becomes a problem if the GStreamer is slower at popping buffers from the queue than Pipewire processes buffers.
- At one point, every Pipewire buffer will be on the queue.
- As GStreamer returns buffers to Pipewire using reference counting, but adds a reference for every occurrence on the queue, GStreamer will never return buffers to Pipewire if the buffer is still on the queue.
- Pipewire will start reusing buffers that have not been returned from GStreamer. This puts a new frame into the Pipewire buffer data and adds the buffer to the queue again. Now the buffer is on the queue twice with the same data but in different positions.
GStreamer doesn't have a chance to process the frames in the correct order or even to return the buffers that are not needed anymore to Pipewire.
The issue can be reproduced with the Pipewire v4l2-source and the GStreamer pipewiresrc. If the application delays the switch to the PLAYING state, because Pipewire starts processing buffers before that, but GStreamer won't take the buffers.
I think that this queue must be modified that a Pipewire buffer can occur on the queue only once. Thus, all occurrences of a buffer on the queue should be dropped when Pipewire processes a buffer. This solves the issue of buffers being on the queue in the wrong order.
Second, the queue must be limited to less than the number of negotiated buffers. This ensures that Pipewire always has enough buffers that can be processed. I'm not sure how long this queue should be, but maybe half of the number of buffers.
I am also not sure, what to do if GStreamer took buffers from the queue but queues it somewhere else (without copying in the pipewiresrc). I guess that it is the responsibility of the application to prevent this from happening.