• Piotr Brzeziński's avatar
    vtenc: Fix deadlock after GST_FLOW_ERROR is received on frame push · 9cbe9a52
    Piotr Brzeziński authored and GStreamer Marge Bot's avatar GStreamer Marge Bot committed
    This was easy to trigger when testing with e.g. vtenc ! vtdec ! glimagesink and closing the sink via window button,
    causing GST_FLOW_ERROR to be received by the output loop, stopping it with the queue still full. This made the
    enqueue_buffer() callback to lock waiting for space in our queue, while handle_frame() was waiting for the internal
    VideoToolbox queue to free up, so that VTCompressionSessionEncodeFrame could finish. As the output loop was not
    running, both functions waited forever.
    Fixed by 1) immediately emptying our queue when GST_FLOW_ERROR is received (like we already did with _FLUSHING)
    and 2) unconditionally setting the flushing flag in finish_encoding() when it sees the output loop stopped because
    of GST_FLOW_ERROR, so that enqueue_buffer() will immediately discard any new frames coming out of VideoToolbox.
    Both of those make sure we never run into the both-queues-full scenario.
    Part-of: <!5303>