Crash when using rtsp client with TLS
Describe your issue
We gather multiple streams to the same device using rtsp with openssl TLS. Our process crashes often when we attempt to close all streams and restart them, but only when we are using TLS.
The errors we receive from the coredumps are often "unsorted double linked list corrupted".
I've been able to get more information using valgrind with 7 streams. Please view "Addition information" to view the stacktrace from valgrind.
I'm unable to provide exact details on how to reproduce the issue, but I'll attempt to answer any questions that you might have. I don't know what I can try to give a better description.
We have currently been using gstreamer version 1.22.0 and 1.22.2.
Expected Behavior
Streams should be closing without memory issues.
Observed Behavior
The process crashes because of invalid read.
Setup
- Operating System: GNU/Linux
- Device: Embedded aarch64 device
- GStreamer Version: 1.22.0 and 1.22.2
- Command line: Not available.
Steps to reproduce the bug
I'm not sure on how to reproduce using only gstreamer tools, but this is essentially what we do:
- Setup multiple streams to one process using rtsp client with openssl TLS:
- Restart the streams in loop until the crash appears.
- A crash can appear between a few minutes to a few hours.
How reproducible is the bug?
Intermittent: it can happen after a few minutes up to multiple hours. I'm assuming this could be a race condition caused by unprotected shared data.
Additional Information
==1088346==
==1088346== Invalid read of size 4
==1088346== at 0x4BF76F8: g_wakeup_signal (gwakeup.c:232)
==1088346== by 0x4BA3E27: g_source_set_ready_time (gmain.c:2023)
==1088346== by 0x4E2556B: cancellable_source_cancelled (gcancellable.c:688)
==1088346== by 0x4AC7C0F: _g_closure_invoke_va (gclosure.c:893)
==1088346== by 0x4AE1A93: g_signal_emit_valist (gsignal.c:3406)
==1088346== by 0x4AE1C97: g_signal_emit (gsignal.c:3553)
==1088346== by 0x4E25E2F: g_cancellable_cancel (gcancellable.c:513)
==1088346== by 0x51B4D47: gst_rtsp_connection_flush (gstrtspconnection.c:3118)
==1088346== by 0x8779E23: gst_rtspsrc_connection_flush (gstrtspsrc.c:5288)
==1088346== by 0x8779FEB: gst_rtspsrc_loop_send_cmd (gstrtspsrc.c:6171)
==1088346== by 0x8782E33: gst_rtspsrc_change_state (gstrtspsrc.c:9295)
==1088346== by 0x504C913: gst_element_change_state (gstelement.c:3083)
==1088346== by 0x504CB07: gst_element_set_state_func (gstelement.c:3037)
==1088346== by 0x502F257: gst_bin_element_set_state (gstbin.c:2581)
==1088346== by 0x502F257: gst_bin_change_state_func (gstbin.c:2923)
==1088346== by 0x506B6A7: gst_pipeline_change_state (gstpipeline.c:529)
==1088346== by 0x504C913: gst_element_change_state (gstelement.c:3083)
==1088346== by 0x504CB07: gst_element_set_state_func (gstelement.c:3037)
==1088346== by 0x16D63F: pipeline_info_unref (cache.c:658)
==1088346== by 0x16ECB3: cache_info_free (cache.c:1197)
==1088346== by 0x16ECB3: cache_info_free (cache.c:1174)
==1088346== by 0x1681C3: base_buffer_stop_caching_unlocked (basebuffer.c:1139)
==1088346== by 0x1681C3: base_buffer_set_state_unlocked (basebuffer.c:2411)
==1088346== by 0x16A86F: base_buffer_message_handler (basebuffer.c:1033)
==1088346== by 0x503784B: gst_bus_source_dispatch (gstbus.c:821)
==1088346== by 0x4BA63FB: g_main_dispatch (gmain.c:3417)
==1088346== by 0x4BA63FB: g_main_context_dispatch (gmain.c:4135)
==1088346== by 0x4BA6787: g_main_context_iterate.constprop.0 (gmain.c:4211)
==1088346== by 0x4BA6B27: g_main_loop_run (gmain.c:4411)
==1088346== by 0x165D1B: loop_func (basebuffer.c:2762)
==1088346== by 0x4BD1E67: g_thread_proxy (gthread.c:827)
==1088346== by 0x5410657: start_thread (pthread_create.c:442)
==1088346== by 0x547765B: thread_start (clone.S:79)
==1088346== Address 0xa239674 is 4 bytes inside a block of size 8 free'd
==1088346== at 0x4867FD8: free (vg_replace_malloc.c:872)
==1088346== by 0x4BA45BF: g_main_context_unref (gmain.c:636)
==1088346== by 0x8BF1717: g_tls_bio_wait_available (gtlsbio.c:539)
==1088346== by 0x8BECB9B: perform_openssl_io (gtlsconnection-openssl.c:341)
==1088346== by 0x8BED28B: g_tls_connection_openssl_read (gtlsconnection-openssl.c:926)
==1088346== by 0x8BF6663: g_tls_connection_base_read (gtlsconnection-base.c:2057)
==1088346== by 0x8BF77E3: g_tls_input_stream_read (gtlsinputstream.c:82)
==1088346== by 0x4E5A5EF: g_input_stream_read (ginputstream.c:198)
==1088346== by 0x51AFBD7: fill_raw_bytes (gstrtspconnection.c:1426)
==1088346== by 0x51B0C27: fill_bytes (gstrtspconnection.c:1489)
==1088346== by 0x51B0C27: read_bytes (gstrtspconnection.c:1510)
==1088346== by 0x51B1003: build_next (gstrtspconnection.c:2464)
==1088346== by 0x51B3D33: gst_rtsp_connection_receive_usec (gstrtspconnection.c:2797)
==1088346== by 0x8785E9B: gst_rtspsrc_connection_receive (gstrtspsrc.c:2804)
==1088346== by 0x8785E9B: gst_rtspsrc_loop_interleaved (gstrtspsrc.c:5678)
==1088346== by 0x8785E9B: gst_rtspsrc_loop (gstrtspsrc.c:6215)
==1088346== by 0x8785E9B: gst_rtspsrc_thread (gstrtspsrc.c:9177)
==1088346== by 0x508DEFB: gst_task_func (gsttask.c:384)
==1088346== by 0x4BD29C7: g_thread_pool_thread_proxy (gthreadpool.c:354)
==1088346== by 0x4BD1E67: g_thread_proxy (gthread.c:827)
==1088346== by 0x5410657: start_thread (pthread_create.c:442)
==1088346== by 0x547765B: thread_start (clone.S:79)
==1088346== Block was alloc'd at
==1088346== at 0x486551C: malloc (vg_replace_malloc.c:381)
==1088346== by 0x4BAC5E7: g_malloc (gmem.c:125)
==1088346== by 0x4BC55F7: g_slice_alloc (gslice.c:1072)
==1088346== by 0x4BF75BB: g_wakeup_new (gwakeup.c:141)
==1088346== by 0x4BA2E4F: g_main_context_new_with_flags (gmain.c:733)
==1088346== by 0x8BF165F: g_tls_bio_wait_available (gtlsbio.c:495)
==1088346== by 0x8BECB9B: perform_openssl_io (gtlsconnection-openssl.c:341)
==1088346== by 0x8BED28B: g_tls_connection_openssl_read (gtlsconnection-openssl.c:926)
==1088346== by 0x8BF6663: g_tls_connection_base_read (gtlsconnection-base.c:2057)
==1088346== by 0x8BF77E3: g_tls_input_stream_read (gtlsinputstream.c:82)
==1088346== by 0x4E5A5EF: g_input_stream_read (ginputstream.c:198)
==1088346== by 0x51AFBD7: fill_raw_bytes (gstrtspconnection.c:1426)
==1088346== by 0x51B0C27: fill_bytes (gstrtspconnection.c:1489)
==1088346== by 0x51B0C27: read_bytes (gstrtspconnection.c:1510)
==1088346== by 0x51B1003: build_next (gstrtspconnection.c:2464)
==1088346== by 0x51B3D33: gst_rtsp_connection_receive_usec (gstrtspconnection.c:2797)
==1088346== by 0x8785E9B: gst_rtspsrc_connection_receive (gstrtspsrc.c:2804)
==1088346== by 0x8785E9B: gst_rtspsrc_loop_interleaved (gstrtspsrc.c:5678)
==1088346== by 0x8785E9B: gst_rtspsrc_loop (gstrtspsrc.c:6215)
==1088346== by 0x8785E9B: gst_rtspsrc_thread (gstrtspsrc.c:9177)
==1088346== by 0x508DEFB: gst_task_func (gsttask.c:384)
==1088346== by 0x4BD29C7: g_thread_pool_thread_proxy (gthreadpool.c:354)
==1088346== by 0x4BD1E67: g_thread_proxy (gthread.c:827)
==1088346== by 0x5410657: start_thread (pthread_create.c:442)
==1088346== by 0x547765B: thread_start (clone.S:79)
==1088346==