Skip to content

rtpjpegpay: Passing JPEG images through 'rtpjpegpay' and 'rtpjpegdepay' results in corrupt output on Windows

Details

On Windows, version 1.18.4:

This pipeline (on both Windows and Linux) results in a corrupted output:

filesrc location=C:/test_in.jpeg ! jpegparse ! rtpjpegpay ! rtpjpegdepay ! filesink location=C:/test_out.jpeg

This pipeline (possibly only on Windows) also results in a corrupted output:

videotestsrc ! jpegenc ! rtpjpegpay ! rtpjpegdepay ! jpegdec ! autovideosink

In either of the above pipelines, it's easy to confirm that the payloader and/or depayloader are responsible for this by simply removing them from the pipeline. I am confident that it is the payloader and/or depayloader that are at fault based on several hours of trying different combinations of encoders, decoders, and other methods of sending and receiving JPEG data, also using different pixel formats.

Result Without Payloader and Depayloader Result with Payloader and Depayloader
capture2_test_pay2 capture2_test_pay

You can see that the result with the payloader and depayloader (the "corrupted" image):

  1. has the wrong colors (too bright),
  2. is shifted over to the left (and wraps back around on the right), and
  3. has a small corrupted rectangle in the bottom right corner.

Certain images may only exhibit the first problem.

Research

I haven't been able to determine what specifically might be causing this, but I have made three important observations about the image that results from passing through the payloader and depayloader:

  1. Any APP0 headers from the original image (the "uncorrupted" image) will be missing.
  2. Each color component has an ID 1 less than those of the uncorrupted image (0, 1, 2 instead of 1, 2, 3).
  3. Two consecutive bytes are missing from the SOS section (this can occur in different places depending on the image)

It seems that fixing the third issue on any given image (adding back in the two missing consecutive bytes) is enough to fix the result image so that it is identical to the uncorrupted image.

I made these observations:

  • using xxd via my MinGW terminal to convert JPEG images into hexadecimal (and back),
  • comparing hexadecimal dumps of images using a diff tool,
  • examining RTP packets prepared by the payloading using Wireshark, and
  • comparing JPEG header contents using the tool at https://cyber.meme.tips/jpdump/
Edited by Tim-Philipp Müller