vtenc: "Output state was not configured"

Submitted by Ilya Konstantinov

Link to original bug (#747546)

Description

gst_vtenc_finish might be called before gst_vtenc_encode_frame ever manages to encode a single frame, i.e.

i = 0;
while (g_async_queue_length (self->cur_outframes) > 0) {
GstVideoCodecFrame *outframe = g_async_queue_try_pop (self->cur_outframes);

/* Try to renegotiate once */  
if (i == 0) {  
  meta = gst_buffer_get_core_media_meta (outframe->output_buffer);  
  if (!gst_vtenc_negotiate_downstream (self, meta->sample_buf)) {  

       `^``^` this code might never run before gst_vtenc_finish  

In such case, VT's queue will be flushed and gst_video_encoder_finish_frame will be called for every frame, but negotiation will never happen, and therefore:

ERROR videoencoder gstvideoencoder.c:2033:GstFlowReturn gst_video_encoder_finish_frame(GstVideoEncoder *, GstVideoCodecFrame *):<vtenc_h264-0> Output state was not configured