matroska: segfault / potential heap overflow in zlib decoding

Describe the vulnerability

Integer overflow while decoding zlib encoded data in gst_matroska_decompress_data.

The given crashing POC zlib-decode-overflow-1.mkv has a block that when zlib decompressed is 0x100000000.

This overflows the new_size integer here: https://gitlab.freedesktop.org/gstreamer/gstreamer/-/blob/main/subprojects/gst-plugins-good/gst/matroska/matroska-read-common.c#L117

One potential root cause is that the zstream data counters are 64-bit: https://gitlab.freedesktop.org/gstreamer/gstreamer/-/blob/main/subprojects/gst-plugins-good/gst/matroska/matroska-read-common.c#L105

while 32-bit integers are used to store the size in gst_matroska_decompress_data https://gitlab.freedesktop.org/gstreamer/gstreamer/-/blob/main/subprojects/gst-plugins-good/gst/matroska/matroska-read-common.c#L80

Expected Behavior

Not segfault.

Observed Behavior

Segfault

Setup

• Operating System: Ubuntu 20.04.4
• Device: Computer
• GStreamer Version: 1.16.2
• Command line: gst-play-1.0 ./zlib-decode-overflow-1.mkv

Steps to reproduce the bug

1. Download zlib-decode-overflow-1.mkv
2. Run gst-play-1.0 ./zlib-decode-overflow-1.mkv (takes about 8 seconds to trigger on my system)

How reproducible is the bug?

Always

Impact

Depending on the libc used, and the underlying OS capabilities, it could be just a segfault or a heap overwrite.

If the libc uses mmap for large chunks, and the OS supports mmap, then it is just a segfault (because the realloc before the integer overflow will use mremap to reduce the size of the chunk, and it will start to write to unmapped memory).

However, if using a libc implementation that does not use mmap, or if the OS does not support mmap while using libc, then this would likely result in a heap overwrite.

Additional Information

Not sure if this qualifies for a CVE, not sure what this project does in the past. Happy to discuss.

Thanks!