GitLab

Issue:

Using the function gst_h264_create_sei_memory_internal of the gsth264parser generates a overflow of 0xFF using a payload_size_data bigger than 0XFF.

Cause:

gsth264parser has a while process to write 0xFF in the SEI message and subtract 0XFF from the payload_size_data, but right now the while is adding 0XFF to the payload_size_data and the while never ends.

Proposed fix:

Change the subtraction in the while to subtract 0XFF correctly. To do this it's possible to apply the following changes, the following changes also fix the same issue for the payload_type_data.

diff --git a/gst-libs/gst/codecparsers/gsth264parser.c b/gst-libs/gst/codecparsers/gsth264parser.c
index 1c40b6517..22b1e4bca 100644
--- a/gst-libs/gst/codecparsers/gsth264parser.c
+++ b/gst-libs/gst/codecparsers/gsth264parser.c
@@ -3120,14 +3120,14 @@ gst_h264_create_sei_memory_internal (guint8 nal_prefix_size,
     /* write payload type bytes */
     while (payload_type_data >= 0xff) {
       WRITE_UINT8 (&nw, 0xff, 8);
-      payload_type_data -= -0xff;
+      payload_type_data -= 0xff;
     }
     WRITE_UINT8 (&nw, payload_type_data, 8);
 
     /* write payload size bytes */
     while (payload_size_data >= 0xff) {
       WRITE_UINT8 (&nw, 0xff, 8);
-      payload_size_data -= -0xff;
+      payload_size_data -= 0xff;
     }
     WRITE_UINT8 (&nw, payload_size_data, 8);

Should I try to make a pull request?