Skip to content

gsth264parser: Fix handling of NALs with emulation byte set

Marek Vasut requested to merge marex/gst-plugins-bad:v4l2slh264dec/master/nal-hwdec-emulation-bytes into master

In case a set of NALs with emulation_prevention_three_byte is decoded using hardware decoder like Hantro G1, wrong struct v4l2_ctrl_h264_decode_params .dec_ref_pic_marking_bit_size is passed into the kernel, which results in decoding artifacts. Subtract the number of emulation three bytes from the .dec_ref_pic_m->bit_size to get the correct bit size and avoid having these artifacts. Apply the exact same fix to slice->pic_order_cnt_bit_size as well.

The following NALs (7, 8, 6, 5) decode with artifacts, .dec_ref_pic_marking_bit_size is set to 10 without this patch.

00000000  00 00 00 01 27 4d 00 20  89 8b 60 3c 04 bf 2e 02  |....'M. ..`<....|
00000010  d4 18 04 18 c0 c0 01 77  00 00 5d c1 7b df 05 00  |.......w..].{...|
00000020  00 00 01 28 ee 1f 20 00  00 01 06 05 10 b9 ed b9  |...(.. .........|
00000030  30 5d 21 4b 71 83 71 2c  10 a3 14 bb 29 80 00 00  |0]!Kq.q,....)...|
00000040  01 25 b8 00 05 00 00 03  03 7f fa 78 1e e7 fd fe  |.%.........x....|
                         ^^^^^^^^^^^^--- emulation 3 byte
00000050  b4 62 7a 31 ff 7d 81 fd  26 d8 62 b6 d6 25 46 ae  |.bz1.}..&.b..%F.|

The following NALs (7, 8, 6, 5) decode fine, .dec_ref_pic_marking_bit_size is set to 2 without this patch.

00000000  00 00 00 01 27 4d 00 20  89 8b 60 3c 04 bf 2e 02  |....'M. ..`<....|
00000010  d4 18 04 18 c0 c0 01 77  00 00 5d c1 7b df 05 00  |.......w..].{...|
00000020  00 00 01 28 ee 1f 20 00  00 01 06 05 10 b9 ed b9  |...(.. .........|
00000030  30 5d 21 4b 71 83 71 2c  10 a3 14 bb 29 80 00 00  |0]!Kq.q,....)...|
00000040  01 25 b8 00 04 c0 00 03  7f fa 78 1e e7 fd fe b4  |.%........x.....|
00000050  62 7a 31 ff 7d 81 fd 26  d8 62 b6 d6 25 46 ae ce  |bz1.}..&.b..%F..|
Edited by Marek Vasut

Merge request reports