Skip to content

vtdec: Fix a deadlock during ProRes playback, handle non-linked gracefully

Sometimes a call to negotiate (and thus drain) can happen from the output loop
(via finish_frame()), which will tell VT to output all internal frames, but that won't succeed
if we happen to decide to wait for the queue to empty (because the loop is waiting for draining to finish and
will not make space in the queue!). This commit adds an override for the queue size limit if we're draining/flushing.

This bug could happen for any formats, but was especially obvious for ProRes, which has dpb_size of 0.
In an early non-linked scenario, this was causing a ton of criticals about the queue array,
because the output callback would still fire for leftover frames that were still being processed by VT
at the time the output loop stopped. This makes sure they're flushed correctly as well.

Also renames gst_vtdec_loop to gst_vtdec_output_loop for consistency with related functions.
Edited by Piotr Brzeziński

Merge request reports