Skip to content

Revert and properly repair "jpegdec: only allow conversions from RGB"

Marek Vasut requested to merge marex/gstreamer:jpegdec-real-fix into main

This first reverts commit 2aa24772. The commit is completely wrong, libjpeg-turbo is perfectly capable of decoding I420 (YUV) to RGB. The test case provided alongside the aforementioned commit passes without this revert because it decodes image of JCS_YCrCb color space, so the new if (clrspc == JCS_RGB) condition is false on that image, and the libjpeg-turbo decoding does not get used. The real bug is hidden by that commit.

The real problem is in the call order of gst_jpeg_dec_prepare_decode() and gst_jpeg_dec_negotiate(). The gst_jpeg_dec_prepare_decode() calls jpeg_start_decompress() which sets up internal state of the libjpeg, however, neither cinfo.out_color_space nor cinfo.raw_data_out are set correctly yet. Those two are set up in gst_jpeg_dec_negotiate() which is called a bit later. Therefore, the real fix is the set up cinfo.out_color_space and cinfo.raw_data_out before calling jpeg_start_decompress().

Pull out peer caps checking code into gst_jpeg_turbo_parse_ext_fmt_convert(). This code is used by libjpeg-turbo extras to determine whether peer is capable of handling buffers into which libjpeg-turbo can directly decode data. This kind of check must be performed before jpeg_start_decompress() is called in gst_jpeg_dec_prepare_decode() as well as in gst_jpeg_dec_negotiate(), hence the common code.

It is imperative that the libjpeg-turbo state is properly initialized before jpeg_start_decompress() is called. Make sure cinfo.out_color_space and cinfo.raw_data_out are set to their final values matching their peer caps before calling jpeg_start_decompress() by calling gst_jpeg_turbo_parse_ext_fmt_convert() first.

Finally, the libjpeg-turbo internal state might not be correctly initialized for the first frame in a stream, pull the frame stride from gstreamer frame metadata instead, which is correct even for the first frame, and which makes this code consistent with the surrounding lines.

Merge request reports