mux: aac/h264 out of sync issue
The muxed file is out of sync when using gstreamer. Tested on x264enc, fdkaacenc/voaacenc, mp4mux/mpegtsmux. FFmpeg produce file in sync. For the ffmpeg the delay is: "avg_av_delay_ms": -1.6
same as in the original file. For the gstreamer the delay is "avg_av_delay_ms": 41.4
(fdkaac) or "avg_av_delay_ms": 32.2
(voaac). See graph below (audio peaks).
- Operating System: ubuntu 20.04
- GStreamer Version: main
The delay is measured using https://github.com/UbiCastTeam/qr-lipsync. Results:
test_sample.mp4 (original file)
{
"duplicated_frames": 0,
"duplicated_frames_percent": 0,
"dropped_frames": 0,
"dropped_frames_percent": 0,
"total_frames": 1800,
"total_beeps": 30,
"avg_real_framerate": 60,
"median_av_delay_ms": -2,
"median_av_delay_frames": 0,
"avg_av_delay_ms": -1.6,
"avg_av_delay_frames": 0,
"av_delay_accel": 0,
"max_delay_ms": -3,
"max_delay_ts": 0.997,
"video_duration": 30,
"audio_duration": 30.059,
"matching_missing": 0
}
ffmpeg -i test_sample.mp4 -c:v libx264 -c:a aac ffmpeg.mp4
:
ffmpeg.mp4
{
"duplicated_frames": 0,
"duplicated_frames_percent": 0,
"dropped_frames": 0,
"dropped_frames_percent": 0,
"total_frames": 1800,
"total_beeps": 30,
"avg_real_framerate": 60,
"median_av_delay_ms": -2,
"median_av_delay_frames": 0,
"avg_av_delay_ms": -1.6,
"avg_av_delay_frames": 0,
"av_delay_accel": 0,
"max_delay_ms": -3,
"max_delay_ts": 0.997,
"video_duration": 30.033,
"audio_duration": 30.079,
"matching_missing": 0
}
gst-launch-1.0 filesrc location=test_sample.mp4 ! decodebin name=d mp4mux name=m ! filesink location=gst_fdk.mp4 d. ! queue ! videoconvert ! x264enc ! h264parse ! queue ! m. d. ! queue ! audioconvert ! fdkaacenc ! aacparse ! queue ! m.
:
gst_fdk.mp4
{
"duplicated_frames": 0,
"duplicated_frames_percent": 0,
"dropped_frames": 0,
"dropped_frames_percent": 0,
"total_frames": 1800,
"total_beeps": 30,
"avg_real_framerate": 60,
"median_av_delay_ms": 41,
"median_av_delay_frames": 2,
"avg_av_delay_ms": 41.4,
"avg_av_delay_frames": 2,
"av_delay_accel": 0,
"max_delay_ms": 45,
"max_delay_ts": 10.045,
"video_duration": 30.033,
"audio_duration": 30.08,
"matching_missing": 0
}
The other tested pipelines:
gst-launch-1.0 filesrc location=test_sample.mp4 ! decodebin name=d mp4mux name=m ! filesink location=gst_voaac.mp4 d. ! queue ! videoconvert ! x264enc ! h264parse ! queue ! m. d. ! queue ! audioconvert ! voaacenc ! aacparse ! queue ! m.
gst-launch-1.0 filesrc location=test_sample.mp4 ! decodebin name=d mpegtsmux name=m ! filesink location=gstfdk_ts.ts d. ! queue ! videoconvert ! x264enc ! h264parse ! queue ! m. d. ! queue ! audioconvert ! fdkaacenc ! aacparse ! queue ! m.
In a simple words. Every single encoding a file will delay audio about 30-40 ms.