vaapih264enc AVC failure on AMD RAVEN
Running vaapih264enc in stream-format=avc mode fails on AMD RAVEN with the Mesa Gallium VA driver:
vainfo: Driver version: Mesa Gallium driver 19.3.1 for AMD Ryzen Embedded V1605B with Radeon Vega Gfx (RAVEN, DRM 3.35.0, 5.4.8, LLVM 9.0.0)
Example pipeline:
gst-launch-1.0 videotestsrc ! vaapih264enc ! video/x-h264,stream-format=avc ! fakesink
The VA-API driver returns VA_ENC_PACKED_HEADER_NONE to a query of VAConfigAttribEncPackedHeaders:
INFO vaapi gstvaapiencoder.c:791:get_packed_headers: supported packed headers: 0x00000000
But gst_vaapiencode_h264_set_config sets base_encode->need_codec_data for avc. As a result, gst_vaapi_encoder_h264_get_codec_data fails because ensure_sequence and ensure_picture never submit SPS/PPS headers into the bitstream, and ensure_output_state then fails to get the codec data:
ERROR vaapiencode gstvaapiencode.c:353:gst_vaapiencode_push_frame: failed to negotiate output state (status 0)
This error triggers for every input frame, and vaapih264enc does not produce any output.