omxh264enc profile negotiation fails with non fixated caps
omxh264enc profile negotiation fails if the caps are not fixed
if trying to use the CABAC entropy encoding the encoder needs to be producing profile "high" or above, if the encoder is given a pixel format that forces this (e.g. NV16 or NV12_10LE32) then everything works. however if given a format supported by main or below such as NV12 we have to set output caps to negotiate the profile.
if the caps contain a list of accepted values such as:
video/x-h264,alignment=nal,profile={ high, high-10, high-4:2:2 }
then the profile is not fed to the omx layer, so the encoder chooses an appropriate profile instead. when this is then fed back the caps are checked and negotiation fails when they don't match.
I believe this happens here: https://gitlab.freedesktop.org/gstreamer/gstreamer/-/blob/main/subprojects/gst-omx/omx/gstomxh264enc.c#L695
Expected Behavior
caps should negotiate the profile to one of the values in the list
Observed Behavior
the caps are rejected once the encoder starts producing buffers: 0:00:07.681852659 368 0x558abecd80 WARN GST_CAPS gstpad.c:5757:pre_eventfunc_check:<vid_enc_caps_0:sink> caps video/x-h264, stream-format=(string)byte-stream, alignment=(string)nal, profile=(string)constrained-baseline, level=(string)3.1, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1, interlace-mode=(string)progressive, colorimetry=(string)bt709 not accepted
Setup
- linux
- Xilinx ZCU106
- 1.20.5
- gst-launch-1.0 videotestsrc ! video/x-raw,width=320,height=240,framerate=24/1,format=NV12 ! omxh264enc entropy-mode=CABAC ! video/x-h264,profile={ high, high-10, high-4:2:2 } ! fakesink
Steps to reproduce the bug
- open terminal
- type
gst-launch-1.0 videotestsrc ! video/x-raw,width=320,height=240,framerate=24/1,format=NV12 ! omxh264enc entropy-mode=CABAC ! video/x-h264,profile={ high, high-10, high-4:2:2 } ! fakesink
How reproducible is the bug?
Always happens for NV12 format if a list of profiles is provided in the caps. if a single profile is provided then negotiation works as expected
Solutions you have tried
adding peercaps = gst_caps_fixate (peercaps);
before gstomxh264enc.c#L694 fixes the issue for my use case, but I'm not convinced it will be correct for all scenarios