webrtcsink: Errors while generating debug graph during shutdown of pipeline
Describe your issue
During implementation of !1442 (merged) I found that when GST_DEBUG_DUMP_DOT_DIR
is set, using GStreamer 1.22 and gst-plugins-rs 0.11 branch, the livekitwebrtcsink will occasionally cause lots of glib assertion errors to get logged when the pipeline is shutting down.
Expected Behavior
The pipeline should always shut down cleanly and graphviz dot files should be written to disk.
Observed Behavior
There is a long list of assertions failing while a dot file is being generated:
Freeing pipeline ...
(gst-launch-1.0:839730): GStreamer-CRITICAL **: 10:29:15.393: gst_mini_object_ref: assertion 'mini_object != NULL' failed
(gst-launch-1.0:839730): GStreamer-CRITICAL **: 10:29:15.393: gst_mini_object_unref: assertion 'mini_object != NULL' failed
(gst-launch-1.0:839730): GStreamer-CRITICAL **: 10:29:15.393: gst_caps_features_is_equal: assertion 'features1 != NULL' failed
(gst-launch-1.0:839730): GStreamer-CRITICAL **: 10:29:15.393: gst_mini_object_ref: assertion 'mini_object != NULL' failed
(gst-launch-1.0:839730): GStreamer-CRITICAL **: 10:29:15.393: gst_caps_is_any: assertion 'GST_IS_CAPS (caps)' failed
Setup
- Operating System: Fedora Linux 39
- Device: Computer
- gst-plugins-rs Version: 0.11 (5bba2f78)
- GStreamer Version: 1.22.9+ (gstreamer@23b88fe0)
- Command line:
export GST_DEBUG_DUMP_DOT_DIR="$PWD/dot"
mkdir -p "$GST_DEBUG_DUMP_DOT_DIR"
gst-launch-1.0 livekitwebrtcsink name=sink \
signaller::ws-url=ws://localhost:7880 \
signaller::api-key=devkey \
signaller::secret-key=secret \
signaller::room-name=test \
video-caps='video/x-vp8' \
videotestsrc is-live=1 ! video/x-raw,framerate=30/1,width=320,height=180 ! queue ! sink. \
videotestsrc is-live=1 ! video/x-raw,framerate=30/1,width=640,height=360 ! queue ! sink. \
videotestsrc is-live=1 ! video/x-raw,framerate=30/1,width=1280,height=720 ! queue ! sink.
Steps to reproduce the bug
- Build specified versions of GStreamer and gst-plugins-rs
- Run LiveKit server with
--dev
flag to allow developer credentials specified above to be accepted - type
command
and press ctrl-c a few seconds after starting. Repeat as necessary.
How reproducible is the bug?
Intermittent, about 1/10 attempts have triggered the bug for me. So far I have only ever experienced the bug when using the livekitwebrtcsink, not the plain webrtcsink with the GST reference signalling server.
Screenshots if relevant
N/A
Solutions you have tried
N/A
Related non-duplicate issues
N/A
Additional Information
Stack trace:
#0 g_logv (log_domain=0x7ff06a7dd1d8 "GStreamer", log_level=G_LOG_LEVEL_CRITICAL, format=, args=args@entry=0x7ff0428f2f60) at ../glib/gmessages.c:1423 #1 0x00007ff06a5e9463 in g_log (log_domain=log_domain@entry=0x7ff06a7dd1d8 "GStreamer", log_level=log_level@entry=G_LOG_LEVEL_CRITICAL, format=format@entry=0x7ff06a651280 "%s: assertion '%s' failed") at ../glib/gmessages.c:1461 #2 0x00007ff06a5ea6dd in g_return_if_fail_warning (log_domain=log_domain@entry=0x7ff06a7dd1d8 "GStreamer", pretty_function=pretty_function@entry=0x7ff06a7e3620 <__func__.4> "gst_caps_features_is_any", expression=expression@entry=0x7ff06a7e2991 "features != NULL") at ../glib/gmessages.c:2930 #3 0x00007ff06a746e6e in gst_caps_features_is_any (features=features@entry=0x0) at ../subprojects/gstreamer/gst/gstcapsfeatures.c:766 #4 0x00007ff06a743e1a in gst_caps_is_subset (subset=subset@entry=0x7ff00c0391c0, superset=superset@entry=0x7ff00c019e60) at ../subprojects/gstreamer/gst/gstcaps.c:1348 #5 0x00007ff06a7441e7 in gst_caps_is_equal (caps1=caps1@entry=0x7ff00c0391c0, caps2=caps2@entry=0x7ff00c019e60) at ../subprojects/gstreamer/gst/gstcaps.c:1473 #6 0x00007ff06a74f740 in debug_dump_element_pad_link (pad=0x7ff00c02b870, element=0x7ff00c02b150, details=GST_DEBUG_GRAPH_SHOW_VERBOSE, str=0x7ff00c1074d0, indent=) at ../subprojects/gstreamer/gst/gstdebugutils.c:502 #7 0x00007ff06a750477 in debug_dump_element (bin=bin@entry=0x7ff00c034c90, details=details@entry=GST_DEBUG_GRAPH_SHOW_VERBOSE, str=str@entry=0x7ff00c1074d0, indent=indent@entry=4) at ../subprojects/gstreamer/gst/gstdebugutils.c:714 #8 0x00007ff06a7502fa in debug_dump_element (bin=bin@entry=0x7ff00c0c7030, details=details@entry=GST_DEBUG_GRAPH_SHOW_VERBOSE, str=str@entry=0x7ff00c1074d0, indent=indent@entry=3) at ../subprojects/gstreamer/gst/gstdebugutils.c:694 #9 0x00007ff06a7502fa in debug_dump_element (bin=bin@entry=0x7ff030048f40, details=details@entry=GST_DEBUG_GRAPH_SHOW_VERBOSE, str=str@entry=0x7ff00c1074d0, indent=indent@entry=2) at ../subprojects/gstreamer/gst/gstdebugutils.c:694 #10 0x00007ff06a7502fa in debug_dump_element (bin=bin@entry=0x7ff030066af0, details=details@entry=GST_DEBUG_GRAPH_SHOW_VERBOSE, str=str@entry=0x7ff00c1074d0, indent=indent@entry=1) at ../subprojects/gstreamer/gst/gstdebugutils.c:694 #11 0x00007ff06a750580 in gst_debug_bin_to_dot_data (bin=bin@entry=0x7ff030066af0, details=details@entry=GST_DEBUG_GRAPH_SHOW_VERBOSE) at ../subprojects/gstreamer/gst/gstdebugutils.c:829 #12 0x00007ff06a750748 in gst_debug_bin_to_dot_file (bin=bin@entry=0x7ff030066af0, details=details@entry=GST_DEBUG_GRAPH_SHOW_VERBOSE, file_name=, file_name@entry=0x7ff030058680 "0.00.01.232958865-webrtcsink-session-unique-Paused-to-Ready") at ../subprojects/gstreamer/gst/gstdebugutils.c:873 #13 0x00007ff06a7509aa in gst_debug_bin_to_dot_file_with_ts (bin=0x7ff030066af0, details=GST_DEBUG_GRAPH_SHOW_VERBOSE, file_name=) at ../subprojects/gstreamer/gst/gstdebugutils.c:920 #14 0x00007ff05b88a993 in gstreamer::auto::functions::debug_bin_to_dot_file_with_ts (bin=, details=..., file_name=...) at /var/home/jyelloz/.cargo/git/checkouts/gstreamer-rs-79e52a2d27eb91a3/8c286fa/gstreamer/src/auto/functions.rs:53 #15 0x00007ff05b7d6009 in gstreamer::bin::GstBinExtManual::debug_to_dot_file_with_ts (self=0x7ff0428f3558, details=..., file_name=...) at /var/home/jyelloz/.cargo/git/checkouts/gstreamer-rs-79e52a2d27eb91a3/8c286fa/gstreamer/src/bin.rs:187 #16 gstrswebrtc::webrtcsink::imp::{impl#16}::start_session::{async_block#5} () at net/webrtc/src/webrtcsink/imp.rs:2494 #17 0x00007ff05b7c08ef in tokio::runtime::task::core::{impl#6}::poll::{closure#0}> (ptr=0x7ff03006e130) at /var/home/jyelloz/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/task/core.rs:328 #18 tokio::loom::std::unsafe_cell::UnsafeCell>::with_mut, core::task::poll::Poll<()>, tokio::runtime::task::core::{impl#6}::poll::{closure_env#0}>> (self=0x7ff03006e130, f=...) at /var/home/jyelloz/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/loom/std/unsafe_cell.rs:16 #19 tokio::runtime::task::core::Core>::poll> (self=0x7ff03006e120, cx=...) at /var/home/jyelloz/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/task/core.rs:317 #20 0x00007ff05b81972a in tokio::runtime::task::harness::poll_future::{closure#0}> () at /var/home/jyelloz/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/task/harness.rs:485 #21 core::panic::unwind_safe::{impl#23}::call_once, tokio::runtime::task::harness::poll_future::{closure_env#0}>> (self=...) at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/panic/unwind_safe.rs:272 #22 std::panicking::try::do_call>>, core::task::poll::Poll<()>> (data=) at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552 #23 std::panicking::try, core::panic::unwind_safe::AssertUnwindSafe>>> (f=...) at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516 #24 std::panic::catch_unwind>>, core::task::poll::Poll<()>> (f=...) at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142 #25 tokio::runtime::task::harness::poll_future> (core=0x7ff03006e120, cx=...) at /var/home/jyelloz/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/task/harness.rs:473 #26 tokio::runtime::task::harness::Harness>::poll_inner> (self=) at /var/home/jyelloz/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/task/harness.rs:208 #27 tokio::runtime::task::harness::Harness>::poll> (self=...) at /var/home/jyelloz/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/task/harness.rs:153 #28 0x00007ff05c184466 in tokio::runtime::task::raw::RawTask::poll (self=...) at src/runtime/task/raw.rs:201 #29 tokio::runtime::task::LocalNotified>::run> (self=...) at src/runtime/task/mod.rs:416 #30 tokio::runtime::scheduler::multi_thread::worker::{impl#1}::run_task::{closure#0} () at src/runtime/scheduler/multi_thread/worker.rs:576 #31 tokio::runtime::coop::with_budget, ()>, tokio::runtime::scheduler::multi_thread::worker::{impl#1}::run_task::{closure_env#0}> (budget=..., f=...) at src/runtime/coop.rs:107 #32 tokio::runtime::coop::budget, ()>, tokio::runtime::scheduler::multi_thread::worker::{impl#1}::run_task::{closure_env#0}> (f=...) at src/runtime/coop.rs:73 #33 tokio::runtime::scheduler::multi_thread::worker::Context::run_task (self=0x7ff0428f3a78, task=..., core=) at src/runtime/scheduler/multi_thread/worker.rs:575 #34 0x00007ff05c183218 in tokio::runtime::scheduler::multi_thread::worker::Context::run (self=0x7ff0428f3a78, core=0x7ff050001160) at src/runtime/scheduler/multi_thread/worker.rs:526 #35 0x00007ff05c1882ec in tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure#0} () at src/runtime/scheduler/multi_thread/worker.rs:491 #36 tokio::runtime::context::scoped::Scoped::set (self=0x7ff030000ca8, t=, f=...) at src/runtime/context/scoped.rs:40 #37 0x00007ff05c180754 in tokio::runtime::context::set_scheduler::{closure#0}<(), tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure_env#0}> (c=) at src/runtime/context.rs:176 #38 std::thread::local::LocalKey::try_with, ()> (self=, f=...) at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/thread/local.rs:270 #39 std::thread::local::LocalKey::with, ()> (self=, f=...) at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/thread/local.rs:246 #40 0x00007ff05c17b57c in tokio::runtime::context::set_scheduler<(), tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure_env#0}> (v=0x7ff0428f3a70, f=...) at src/runtime/context.rs:176 #41 tokio::runtime::scheduler::multi_thread::worker::run::{closure#0} () at src/runtime/scheduler/multi_thread/worker.rs:486 #42 tokio::runtime::context::runtime::enter_runtime (handle=, allow_block_in_place=, f=...) at src/runtime/context/runtime.rs:65 #43 0x00007ff05c1829d0 in tokio::runtime::scheduler::multi_thread::worker::run (worker=...) at src/runtime/scheduler/multi_thread/worker.rs:478 #44 0x00007ff05c1634e2 in tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure#0} () at src/runtime/scheduler/multi_thread/worker.rs:447 #45 tokio::runtime::blocking::task::{impl#2}::poll (self=..., _cx=) at src/runtime/blocking/task.rs:42 #46 0x00007ff05c1609ca in tokio::runtime::task::core::{impl#6}::poll::{closure#0}, tokio::runtime::blocking::schedule::BlockingSchedule> (ptr=0x7ff05000d728) at src/runtime/task/core.rs:328 #47 tokio::loom::std::unsafe_cell::UnsafeCell>>::with_mut>, core::task::poll::Poll<()>, tokio::runtime::task::core::{impl#6}::poll::{closure_env#0}, tokio::runtime::blocking::schedule::BlockingSchedule>> (self=0x7ff05000d728, f=...) at src/loom/std/unsafe_cell.rs:16 #48 tokio::runtime::task::core::Core, tokio::runtime::blocking::schedule::BlockingSchedule>::poll, tokio::runtime::blocking::schedule::BlockingSchedule> (self=0x7ff05000d720, cx=...) at src/runtime/task/core.rs:317 #49 0x00007ff05c15f6b5 in tokio::runtime::task::harness::poll_future::{closure#0}, tokio::runtime::blocking::schedule::BlockingSchedule> () at src/runtime/task/harness.rs:485 #50 core::panic::unwind_safe::{impl#23}::call_once, tokio::runtime::task::harness::poll_future::{closure_env#0}, tokio::runtime::blocking::schedule::BlockingSchedule>> (self=...) at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/panic/unwind_safe.rs:272 #51 std::panicking::try::do_call, tokio::runtime::blocking::schedule::BlockingSchedule>>, core::task::poll::Poll<()>> (data=) at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552 #52 std::panicking::try, core::panic::unwind_safe::AssertUnwindSafe, tokio::runtime::blocking::schedule::BlockingSchedule>>> (f=...) at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516 #53 std::panic::catch_unwind, tokio::runtime::blocking::schedule::BlockingSchedule>>, core::task::poll::Poll<()>> (f=...) at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142 #54 tokio::runtime::task::harness::poll_future, tokio::runtime::blocking::schedule::BlockingSchedule> (core=0x440, cx=...) at src/runtime/task/harness.rs:473 #55 0x00007ff05c15e34b in tokio::runtime::task::harness::Harness, tokio::runtime::blocking::schedule::BlockingSchedule>::poll_inner, tokio::runtime::blocking::schedule::BlockingSchedule> (self=) at src/runtime/task/harness.rs:208 #56 0x00007ff05c162a58 in tokio::runtime::task::harness::Harness, tokio::runtime::blocking::schedule::BlockingSchedule>::poll, tokio::runtime::blocking::schedule::BlockingSchedule> (self=...) at src/runtime/task/harness.rs:153 #57 tokio::runtime::task::raw::poll, tokio::runtime::blocking::schedule::BlockingSchedule> (ptr=...) at src/runtime/task/raw.rs:271 #58 0x00007ff05c162f20 in tokio::runtime::task::raw::RawTask::poll (self=...) at src/runtime/task/raw.rs:201 #59 tokio::runtime::task::UnownedTask::run (self=...) at src/runtime/task/mod.rs:453 #60 0x00007ff05c16a251 in tokio::runtime::blocking::pool::Task::run (self=...) at src/runtime/blocking/pool.rs:159 #61 tokio::runtime::blocking::pool::Inner::run (self=0x7ff05000c9c0, worker_thread_id=) at src/runtime/blocking/pool.rs:513 #62 0x00007ff05c16eb1d in tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure#0} () at src/runtime/blocking/pool.rs:471 #63 std::sys_common::backtrace::__rust_begin_short_backtrace (f=...) at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154 #64 0x00007ff05c16f55c in std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure#0} () at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/thread/mod.rs:529 #65 core::panic::unwind_safe::{impl#23}::call_once<(), std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}> (self=) at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/panic/unwind_safe.rs:272 #66 std::panicking::try::do_call>, ()> (data=) at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552 #67 std::panicking::try<(), core::panic::unwind_safe::AssertUnwindSafe>> (f=...) at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516 #68 std::panic::catch_unwind>, ()> (f=) at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142 #69 std::thread::{impl#0}::spawn_unchecked_::{closure#1} () at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/thread/mod.rs:528 #70 core::ops::function::FnOnce::call_once, ()> () at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250 #71 0x00007ff05c2222a5 in alloc::boxed::{impl#47}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:2007 #72 alloc::boxed::{impl#47}::call_once<(), alloc::boxed::Box, alloc::alloc::Global>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:2007 #73 std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108 #74 0x00007ff06a3d2897 in start_thread (arg=) at pthread_create.c:444 #75 0x00007ff06a45980c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78