Codec / SPS / PPS info missing on multi-video track content on track switch
Describe your issue
On pipelines build for multi-video track containers using independent parser for each track, but with the decoder element present after an input switch element connected to the parsers, the decoder may not have the correct codec info when a track switch occurs. Consequently, video freezes. This can be easily observed when switching between tracks with different resolutions.
Expected Behavior
Decoder has up-to-date info and no video freezes are observed
Observed Behavior
Video freeze is observed due missing up-to-date SPS / PPS info
Setup
- Operating System: Linux / Ubuntu, but should be reproducible with other OSs as well
- Device: Computer / Embedded Device
- GStreamer Version: 1.18, 1.20 (should be reproducible also with other versions)
Steps to reproduce the bug
Attached video-track-switch.tar package contains small app that reproduces the issue. Extract archive and from a terminal run:
./gst-demo combined-nok.mp4
The test samples in this package were generated in the following way:
curl http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4 -o BigBuckBunny.mp4
curl http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ElephantsDream.mp4 -o ElephantsDream.mp4
curl http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/WhatCarCanYouGetForAGrand.mp4 -o WhatCarCanYouGetForAGrand.mp4
ffmpeg -i BigBuckBunny.mp4 -i ElephantsDream.mp4 -fs 25M -c copy -map 0:v -map 0:a -map 1:v combined-ok.mp4
ffmpeg -i BigBuckBunny.mp4 -i WhatCarCanYouGetForAGrand.mp4 -fs 25M -c copy -map 0:v -map 0:a -map 1:v combined-nok.mp4
How reproducible is the bug?
Always (with the specific pipeline config)
Solutions you have tried
Merge request 6293 contains a fix for h264parse and h265parse elements, but other parsers may need to be updated.
Additional Information
The issue was originally reproduced with h264parse element. Following pipeline config allows reproducing issue when a track switch occurs after playback started on one of the video tracks:
input-selector name=is is. ! queue ! avdec_h264 ! videoconvert ! autovideosink \
filesrc location=combined-nok.mp4 ! qtdemux name=demux \
demux.video_0 ! queue ! h264parse ! capsfilter caps=video/x-h264,stream-format=byte-stream ! is.sink_0 \
demux.video_1 ! queue ! h264parse ! capsfilter caps=video/x-h264,stream-format=byte-stream ! is.sink_1