Random freeze and deadlock in ffmpegviddec flush and get_buffer while seeking
Describe your issue
When playing different mp4 videos with players that use gstreamer as backend, seeking through the video randomly leads to a full freeze of the application.
Expected Behavior
Playback continues after seeking
Observed Behavior
Playback stops after seeking and the whole media player becomes unresponsive until killed (window buttons not working)
Setup
- Operating System: Fedora 38
- Device: Computer
- GStreamer Version: 1.22.2
- Command line: affects different media players
Fedora 38 packages
Name : gstreamer1-plugin-libav
Version : 1.22.2
Release : 1.fc38
Name : ffmpeg-libs
Version : 6.0
Release : 9.fc38
It did not happen on Fedora 37 with ffmpeg 5
Steps to reproduce the bug
- Create dummy video:
ffmpeg -f lavfi -i testsrc=rate=60 -t 15 -pix_fmt yuv420p dummy.mp4
- Play dummy video with totem:
totem dummy.mp4
- Randomly press the number and arrow keys (to seek) until playback freezes
How reproducible is the bug?
On my machine it is reproducible within seconds. It takes longer (still <30s) when dumping filtered logs with GST_DEBUG=...:5 GST_DEBUG_FILE=...
. When dumping unfiltered logs (GST_DEBUG=5
), I can not reproduce the behavior.
Related non-duplicate issues
- #2383 (closed), other functions, and is already part of 1.22.2 package in Fedora 38, but ffmpeg 6.0 might be relevant
Additional Information
I attached gdb
on a hanging totem process. Looking at the backtrace of each thread, the deadlock is very obvious:
Thread 22 (LWP 21933 "av:h264:df1"):
#0 futex_wait (private=0, expected=2, futex_word=0x7f64d00ddca0) at ../sysdeps/nptl/futex-internal.h:146
#1 __GI___lll_lock_wait (futex=futex@entry=0x7f64d00ddca0, private=0) at lowlevellock.c:49
#2 0x00007f655a8dfe77 in lll_mutex_lock_optimized (mutex=0x7f64d00ddca0) at pthread_mutex_lock.c:48
#3 ___pthread_mutex_lock (mutex=0x7f64d00ddca0) at pthread_mutex_lock.c:128
#4 0x00007f655b5367ed in g_rec_mutex_lock (mutex=mutex@entry=0x7f64d013f348) at ../glib/gthread-posix.c:397
#5 0x00007f655a67c462 in gst_video_decoder_get_frame (decoder=0x7f64d013f230, frame_number=2166) at ../gst-libs/gst/video/gstvideodecoder.c:4177
#6 0x00007f6518051877 in gst_ffmpegviddec_get_buffer2 () at /lib64/gstreamer-1.0/libgstlibav.so
#7 0x00007f64dce30af3 in ff_get_buffer (avctx=0x7f64d02fd040, frame=0x7f64d03c1740, flags=1) at libavcodec/decode.c:1528
#8 0x00007f64dd0ec12a in thread_get_buffer_internal (flags=1, f=0x7f64d03c1740, avctx=0x7f64d02fd040) at libavcodec/pthread_frame.c:915
[...]
Thread 1 (LWP 21895 "totem"):
#0 0x00007f655a8d91d9 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f64d0200614) at futex-internal.c:57
#1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7f64d0200614, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87
#2 0x00007f655a8d925f in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f64d0200614, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at futex-internal.c:139
#3 0x00007f655a8dbb79 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x0, cond=0x7f64d02005e8) at pthread_cond_wait.c:503
#4 ___pthread_cond_wait (cond=cond@entry=0x7f64d02005e8, mutex=mutex@entry=0x7f64d0200640) at pthread_cond_wait.c:618
#5 0x00007f64dd0ec83c in park_frame_worker_threads (fctx=0x7f64d0200300, thread_count=<optimized out>) at libavcodec/pthread_frame.c:638
#6 0x00007f64dcda2507 in ff_thread_flush (avctx=0x7f64d0143100) at libavcodec/pthread_frame.c:864
#7 avcodec_flush_buffers (avctx=0x7f64d0143100) at libavcodec/avcodec.c:404
#8 0x00007f651804eb82 in gst_ffmpegviddec_flush () at /lib64/gstreamer-1.0/libgstlibav.so
#9 0x00007f655a6c0c94 in gst_video_decoder_flush.isra.0 (dec=0x7f64d013f230, hard=1) at ../gst-libs/gst/video/gstvideodecoder.c:1055
#10 0x00007f655a673418 in gst_video_decoder_sink_event_default (decoder=0x7f64d013f230, event=0x55c1a6c674a0) at ../gst-libs/gst/video/gstvideodecoder.c:1616
#11 0x00007f655a79ca2d in gst_pad_send_event_unchecked (pad=pad@entry=0x7f64d013f6e0, event=event@entry=0x55c1a6c674a0, type=<optimized out>, type@entry=320) at ../gst/gstpad.c:5939
#12 0x00007f655a79d117 in gst_pad_push_event_unchecked (pad=pad@entry=0x7f64d0016a60, event=event@entry=0x55c1a6c674a0, type=<optimized out>, type@entry=GST_PAD_PROBE_TYPE_EVENT_DOWNSTREAM) at ../gst/gstpad.c:5572
#13 0x00007f655a79d956 in gst_pad_push_event (pad=0x7f64d0016a60, event=0x55c1a6c674a0) at ../gst/gstpad.c:5711
[...]
Thread 1, stack 10, gst_video_decoder_sink_event_default
locks the decoder for the flush operation. Simultaneously, Thread 22 tries to get a buffer and needs the decoder lock. Thread 1, park_frame_worker_threads
(in ffmpeg, libavcodec) waits for all running threads, including Thread 22 -> Deadlock.
Attachments
-
gst_debug.log last 2000 lines of
GST_DEBUG=videodecoder:7,libav:7 totem ...
. Notice that the 3rd-last linegetting buffer picture
is not in order - gdb_bt.log Full backtrace of all threads of the hanging totem process.