webrtcsink: segmentation fault of python pipeline when connection is reset
Describe your issue
When an pipeline is started with gst-python and ongoing connection is unexpectedly ended, the python script hangs and pipeline disappears. When the script is killed with Ctrl+C, it crashes with a segmentation fault. The signaler correctly shows a warning about the connection being reset without a closing handshake.
Interestingly enough, this doesn't occur when the pipeline is started with gst-launch. (edit: I have managed to crash a pipeline started with gst-launch when running directly with Ubuntu 23.04)
This condition can happen in the real world when a user quickly starts and stops a stream or in case of a disconnection.
I am using the gstwebrtc-api frontend to test this bug.
Expected Behavior
The connection is closed/deleted and the pipeline continues to operate normally.
Observed Behavior
The signaler closes the connection correctly and displays a warning, but the python script crashes with a segmentation fault.
Setup
- Operating System: Ubuntu 23.04
- Device: Docker container on a computer
- gst-plugins-rs Version: latest from main (commit c350f3c2)
- GStreamer Version: 1.22.2
Steps to reproduce the bug
- Copy the docker file, run.sh and test.py scripts into a new folder (provided in the Additional Information section)
- Build the docker image with
docker build --pull -t webrtc-segfault-test .
- Run the image with
docker run --rm -it -p 9090:9090 -p 8443:8443 --entrypoint bash webrtc-segfault-test
- Run the script with
./run.sh
- Connect to
localhost:9090
in a browser - Wait for it to connect to the signaling server and for the producer to be detected
- Quickly press the producer id at least 5 times to open and close the stream
- The pipeline should crash
- After killing the script with Ctrl+C, a segmentation fault is shown
How reproducible is the bug?
It can easily be reproduced by a curious end user, who presses buttons too quickly
Solutions you have tried
I have made a script to restart the python script in a loop, but it is an ugly workaround.
Unfortunately I don't have the knowledge to debug this segmentation fault and come up with a better solution.
Additional Information
Dockerfile
FROM ubuntu:23.04
# Change default shell used by Docker to bash
SHELL ["/bin/bash", "-c"]
# Install common dependencies and prepare the environment.
RUN apt-get update \
&& apt-get -y install \
vim git curl \
# required for building gst-plugins-rs
build-essential libssl-dev libx264-dev libvpx-dev libopus-dev \
# required for webrtc-sink to work properly
libnice-dev gstreamer1.0-nice \
# gstreamer
libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-good1.0-dev libgstreamer-plugins-bad1.0-dev \
gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav \
gstreamer1.0-tools python3-gst-1.0 \
&& rm -rf /var/lib/apt/lists/*
SHELL ["/bin/bash", "--login", "-c"]
# install nvm
ENV NVM_DIR /usr/local/nvm
RUN curl https://raw.githubusercontent.com/creationix/nvm/v0.30.1/install.sh | bash \
&& source $NVM_DIR/nvm.sh \
&& nvm install 18 \
&& nvm alias default 18 \
&& nvm use default
# install rustup
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | bash -s -- --default-toolchain stable -y
# clone gst-plugins-rs and select a specific commit, to ensure stable builds
RUN git clone https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs.git \
&& cd gst-plugins-rs && git checkout c350f3c2af30c588e6aafb9810b30dafd366032e \
&& rm -r .git
# install frontend dependencies
WORKDIR /gst-plugins-rs/net/webrtc/gstwebrtc-api
RUN source $NVM_DIR/nvm.sh \
&& npm install
# build plugins and signalling server
WORKDIR /gst-plugins-rs/net/webrtc
RUN cargo build --release
WORKDIR /gst-plugins-rs/net/webrtc/signalling
RUN cargo build --release
WORKDIR /gst-plugins-rs/net/rtp
RUN cargo build --release
# copy test scripts
WORKDIR /
COPY run.sh run.sh
COPY test.py test.py
RUN chmod +x run.sh
run.sh
#!/bin/bash
export GST_PLUGIN_PATH=/gst-plugins-rs/target/release:$GST_PLUGIN_PATH
(trap 'kill 0' SIGINT EXIT;
(cd /gst-plugins-rs/ && WEBRTCSINK_SIGNALLING_SERVER_LOG=debug ./target/release/gst-webrtc-signalling-server) &
(cd /gst-plugins-rs/net/webrtc/gstwebrtc-api && npm start) &
(python3 test.py) &
(gst-launch-1.0 webrtcsink name=ws meta="meta,name=gst-launch" videotestsrc ! ws.) &
wait
)
test.py
import gi
gi.require_version("Gst", "1.0")
from gi.repository import Gst
Gst.init(None)
pipeline = Gst.Pipeline.new("pipeline")
source = Gst.ElementFactory.make("videotestsrc", "camera-source")
sink = Gst.ElementFactory.make("webrtcsink", "webrtc")
pipeline.add(source)
pipeline.add(sink)
sink_config = Gst.Structure.new_empty("meta")
sink_config.set_value("name", "gst-python")
sink.set_property("meta", sink_config)
source.link(sink)
pipeline.set_state(Gst.State.PLAYING)
pipeline.get_state(Gst.CLOCK_TIME_NONE)
pipeline.get_bus().timed_pop_filtered(Gst.CLOCK_TIME_NONE, Gst.MessageType.EOS)
Error log:
2023-07-06T16:15:29.459009Z DEBUG ThreadId(05) start_session{producer_id="e4149b2f-30f1-4e55-8452-3c8e82b9255f" consumer_id="29bf1d58-3bfd-4d43-a828-6a73b345a13e"}: gst_plugin_webrtc_signalling::handlers: new
2023-07-06T16:15:29.459039Z INFO ThreadId(05) start_session{producer_id="e4149b2f-30f1-4e55-8452-3c8e82b9255f" consumer_id="29bf1d58-3bfd-4d43-a828-6a73b345a13e"}: gst_plugin_webrtc_signalling::handlers: started a session id=013e849c-2a39-4bc2-83ba-25e6d17682b5 producer_id=e4149b2f-30f1-4e55-8452-3c8e82b9255f consumer_id=29bf1d58-3bfd-4d43-a828-6a73b345a13e
2023-07-06T16:15:29.459052Z DEBUG ThreadId(05) start_session{producer_id="e4149b2f-30f1-4e55-8452-3c8e82b9255f" consumer_id="29bf1d58-3bfd-4d43-a828-6a73b345a13e"}: gst_plugin_webrtc_signalling::handlers: close time.busy=24.7µs time.idle=19.6µs
2023-07-06T16:15:29.461574Z INFO ThreadId(05) gst_plugin_webrtc_signalling::server: Received message Ok(Text("{\"type\":\"peer\",\"sessionId\":\"013e849c-2a39-4bc2-83ba-25e6d17682b5\",\"sdp\":{\"type\":\"offer\",\"sdp\":\"v=0\\r\\no=- 5471119890174355199 0 IN IP4 0.0.0.0\\r\\ns=-\\r\\nt=0 0\\r\\na=ice-options:trickle\\r\\na=group:BUNDLE video0\\r\\nm=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 102 103\\r\\nc=IN IP4 0.0.0.0\\r\\na=setup:actpass\\r\\na=ice-ufrag:Ssstupc9oXienNzSpy6R0864HG1l8XVa\\r\\na=ice-pwd:2KeoNx2pKFrhbG0KBA2fHjYwRqxbYXEb\\r\\na=rtcp-mux\\r\\na=rtcp-rsize\\r\\na=sendonly\\r\\na=rtpmap:96 VP8/90000\\r\\na=rtcp-fb:96 nack\\r\\na=rtcp-fb:96 nack pli\\r\\na=rtcp-fb:96 ccm fir\\r\\na=rtcp-fb:96 transport-cc\\r\\na=extmap:1 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\\r\\na=rtpmap:97 H264/90000\\r\\na=rtcp-fb:97 nack\\r\\na=rtcp-fb:97 nack pli\\r\\na=rtcp-fb:97 ccm fir\\r\\na=rtcp-fb:97 transport-cc\\r\\na=fmtp:97 packetization-mode=1;profile-level-id=42c015;level-asymmetry-allowed=1\\r\\na=rtpmap:98 VP9/90000\\r\\na=rtcp-fb:98 nack\\r\\na=rtcp-fb:98 nack pli\\r\\na=rtcp-fb:98 ccm fir\\r\\na=rtcp-fb:98 transport-cc\\r\\na=rtpmap:99 H265/90000\\r\\na=rtcp-fb:99 nack\\r\\na=rtcp-fb:99 nack pli\\r\\na=rtcp-fb:99 ccm fir\\r\\na=rtcp-fb:99 transport-cc\\r\\na=fmtp:99 sprop-vps=QAEMAf//BAgAAAMAmAgAAAMAAD+VmAk=;sprop-sps=QgEBBAgAAAMAmAgAAAMAAD+QAUEB4ilLKzSSZX/4AAgAC1BgYGBAAAADAEAAAAeC;sprop-pps=RAHBcrRiQA==\\r\\na=rtpmap:100 red/90000\\r\\na=rtpmap:101 ulpfec/90000\\r\\na=rtpmap:102 rtx/90000\\r\\na=fmtp:102 apt=96\\r\\na=rtpmap:103 rtx/90000\\r\\na=fmtp:103 apt=100\\r\\na=ssrc-group:FID 1443830754 1524212528\\r\\na=ssrc:1443830754 msid:user3482338400@host-b7957654 webrtctransceiver1\\r\\na=ssrc:1443830754 cname:user3482338400@host-b7957654\\r\\na=ssrc:1443830754 msid:user3482338400@host-b7957654 webrtctransceiver1\\r\\na=ssrc:1443830754 cname:user3482338400@host-b7957654\\r\\na=ssrc:1443830754 msid:user3482338400@host-b7957654 webrtctransceiver1\\r\\na=ssrc:1443830754 cname:user3482338400@host-b7957654\\r\\na=ssrc:1443830754 msid:user3482338400@host-b7957654 webrtctransceiver1\\r\\na=ssrc:1443830754 cname:user3482338400@host-b7957654\\r\\na=ssrc:1524212528 msid:user3482338400@host-b7957654 webrtctransceiver1\\r\\na=ssrc:1524212528 cname:user3482338400@host-b7957654\\r\\na=mid:video0\\r\\na=fingerprint:sha-256 14:5E:9D:E7:C6:D8:9D:EF:C8:B5:C9:79:88:36:BE:6B:20:E8:3F:A6:5B:FE:00:49:6D:98:21:7A:E8:21:B6:79\\r\\na=rtcp-mux-only\\r\\n\"}}"))
2023-07-06T16:15:29.462796Z INFO ThreadId(15) gst_plugin_webrtc_signalling::server: Received message Ok(Text("{\"type\":\"peer\",\"sessionId\":\"013e849c-2a39-4bc2-83ba-25e6d17682b5\",\"ice\":{\"candidate\":\"candidate:1 1 UDP 2015363327 10.12.166.57 45058 typ host\",\"sdpMLineIndex\":0}}"))
2023-07-06T16:15:29.462815Z INFO ThreadId(15) gst_plugin_webrtc_signalling::server: Received message Ok(Text("{\"type\":\"peer\",\"sessionId\":\"013e849c-2a39-4bc2-83ba-25e6d17682b5\",\"ice\":{\"candidate\":\"candidate:2 1 TCP 1015021823 10.12.166.57 9 typ host tcptype active\",\"sdpMLineIndex\":0}}"))
(gst-launch-1.0:26775): GLib-GIO-CRITICAL **: 16:15:29.462: g_inet_address_mask_to_string: assertion 'G_IS_INET_ADDRESS_MASK (mask)' failed
2023-07-06T16:15:29.462909Z INFO ThreadId(15) gst_plugin_webrtc_signalling::server: Received message Ok(Text("{"type":"peer","sessionId":"013e849c-2a39-4bc2-83ba-25e6d17682b5","ice":{"candidate":"candidate:3 1 TCP 1010827519 10.12.166.57 60847 typ host tcptype passive","sdpMLineIndex":0}}"))
2023-07-06T16:15:29.462921Z INFO ThreadId(15) gst_plugin_webrtc_signalling::server: Received message Ok(Text("{"type":"peer","sessionId":"013e849c-2a39-4bc2-83ba-25e6d17682b5","ice":{"candidate":"candidate:4 1 UDP 2015363583 fe80::7bb7:9af8:3b5b:31a9 52890 typ host","sdpMLineIndex":0}}"))
2023-07-06T16:15:29.462931Z INFO ThreadId(15) gst_plugin_webrtc_signalling::server: Received message Ok(Text("{"type":"peer","sessionId":"013e849c-2a39-4bc2-83ba-25e6d17682b5","ice":{"candidate":"candidate:5 1 TCP 1015022079 fe80::7bb7:9af8:3b5b:31a9 9 typ host tcptype active","sdpMLineIndex":0}}"))
2023-07-06T16:15:29.462942Z INFO ThreadId(15) gst_plugin_webrtc_signalling::server: Received message Ok(Text("{"type":"peer","sessionId":"013e849c-2a39-4bc2-83ba-25e6d17682b5","ice":{"candidate":"candidate:6 1 TCP 1010827775 fe80::7bb7:9af8:3b5b:31a9 50231 typ host tcptype passive","sdpMLineIndex":0}}"))
2023-07-06T16:15:29.508250Z INFO ThreadId(05) gst_plugin_webrtc_signalling::server: Received message Ok(Text("{"type":"peer","sessionId":"013e849c-2a39-4bc2-83ba-25e6d17682b5","sdp":{"type":"answer","sdp":"v=0\r\no=mozilla...THIS_IS_SDPARTA-99.0 5474611296026700615 0 IN IP4 0.0.0.0\r\ns=-\r\nt=0 0\r\na=sendrecv\r\na=fingerprint:sha-256 BA:1E:06:22:18:58:FD:49:99:8E:71:02:7F:71:8A:75:9B:16:36:08:E5:28:8D:EF:56:ED:30:5E:8C:5E:90:1A\r\na=group:BUNDLE video0\r\na=ice-options:trickle\r\na=msid-semantic:WMS *\r\nm=video 9 UDP/TLS/RTP/SAVPF 96 102 97 98\r\nc=IN IP4 0.0.0.0\r\na=recvonly\r\na=extmap:1 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\r\na=fmtp:97 profile-level-id=42e01f;level-asymmetry-allowed=1;packetization-mode=1\r\na=fmtp:96 max-fs=12288;max-fr=60\r\na=fmtp:102 apt=96\r\na=fmtp:98 max-fs=12288;max-fr=60\r\na=ice-pwd:74044475de4ce27aa5114a674446ee03\r\na=ice-ufrag:84994362\r\na=mid:video0\r\na=rtcp-fb:96 nack\r\na=rtcp-fb:96 nack pli\r\na=rtcp-fb:96 ccm fir\r\na=rtcp-fb:96 transport-cc\r\na=rtcp-fb:97 nack\r\na=rtcp-fb:97 nack pli\r\na=rtcp-fb:97 ccm fir\r\na=rtcp-fb:97 transport-cc\r\na=rtcp-fb:98 nack\r\na=rtcp-fb:98 nack pli\r\na=rtcp-fb:98 ccm fir\r\na=rtcp-fb:98 transport-cc\r\na=rtcp-mux\r\na=rtcp-rsize\r\na=rtpmap:96 VP8/90000\r\na=rtpmap:102 rtx/90000\r\na=rtpmap:97 H264/90000\r\na=rtpmap:98 VP9/90000\r\na=setup:active\r\na=ssrc:607865159 cname:{f6a3d5b9-f095-432b-ac21-34b9e9718530}\r\n"}}"))
2023-07-06T16:15:29.537694Z INFO ThreadId(05) gst_plugin_webrtc_signalling::server: Received message Ok(Text("{"type":"peer","sessionId":"013e849c-2a39-4bc2-83ba-25e6d17682b5","ice":{"candidate":"candidate:0 1 UDP 2122252543 40331dd7-caf3-4816-951b-7cae56ec95ea.local 52840 typ host","sdpMid":"video0","sdpMLineIndex":0,"usernameFragment":"84994362"}}"))
2023-07-06T16:15:29.537811Z INFO ThreadId(05) gst_plugin_webrtc_signalling::server: Received message Ok(Text("{"type":"peer","sessionId":"013e849c-2a39-4bc2-83ba-25e6d17682b5","ice":{"candidate":"candidate:2 1 TCP 2105524479 40331dd7-caf3-4816-951b-7cae56ec95ea.local 9 typ host tcptype active","sdpMid":"video0","sdpMLineIndex":0,"usernameFragment":"84994362"}}"))
2023-07-06T16:15:29.537947Z INFO ThreadId(05) gst_plugin_webrtc_signalling::server: Received message Ok(Text("{"type":"peer","sessionId":"013e849c-2a39-4bc2-83ba-25e6d17682b5","ice":{"candidate":"candidate:1 1 UDP 1686052863 192.76.172.198 51206 typ srflx raddr 0.0.0.0 rport 0","sdpMid":"video0","sdpMLineIndex":0,"usernameFragment":"84994362"}}"))
2023-07-06T16:15:29.548015Z INFO ThreadId(05) gst_plugin_webrtc_signalling::server: Received message Ok(Text("{"type":"peer","sessionId":"013e849c-2a39-4bc2-83ba-25e6d17682b5","ice":{"candidate":"candidate:7 1 UDP 1679819007 192.76.172.198 39217 typ srflx raddr 10.12.166.57 rport 45058","sdpMLineIndex":0}}"))
2023-07-06T16:15:29.548038Z INFO ThreadId(05) gst_plugin_webrtc_signalling::server: Received message Ok(Text("{"type":"peer","sessionId":"013e849c-2a39-4bc2-83ba-25e6d17682b5","ice":{"candidate":"candidate:8 1 TCP 847249663 192.76.172.198 9 typ srflx raddr 10.12.166.57 rport 9 tcptype active","sdpMLineIndex":0}}"))
2023-07-06T16:15:29.581633Z INFO ThreadId(15) gst_plugin_webrtc_signalling::server: Received message Ok(Text("{"type":"endSession","sessionId":"013e849c-2a39-4bc2-83ba-25e6d17682b5"}"))
2023-07-06T16:15:29.581698Z DEBUG ThreadId(15) end_session{peer_id="29bf1d58-3bfd-4d43-a828-6a73b345a13e" session_id="013e849c-2a39-4bc2-83ba-25e6d17682b5"}: gst_plugin_webrtc_signalling::handlers: new
2023-07-06T16:15:29.581727Z DEBUG ThreadId(15) end_session{peer_id="29bf1d58-3bfd-4d43-a828-6a73b345a13e" session_id="013e849c-2a39-4bc2-83ba-25e6d17682b5"}: gst_plugin_webrtc_signalling::handlers: close time.busy=10.5µs time.idle=19.4µs
2023-07-06T16:15:29.581811Z INFO ThreadId(05) gst_plugin_webrtc_signalling::server: Received message Ok(Text("{"type":"peer","sessionId":"013e849c-2a39-4bc2-83ba-25e6d17682b5","ice":{"candidate":"candidate:9 1 TCP 843055359 192.76.172.198 60847 typ srflx raddr 10.12.166.57 rport 60847 tcptype passive","sdpMLineIndex":0}}"))
ERROR: from element /GstPipeline:pipeline0/GstWebRTCSink:ws: GStreamer encountered a general stream error.
Additional debug info:
net/webrtc/src/webrtcsink/imp.rs(1456): gstrswebrtc::webrtcsink::imp::BaseWebRTCSink::connect_signaller::{{closure}}::{{closure}} (): /GstPipeline:pipeline0/GstWebRTCSink:ws:
Signalling error: Error message from server: Session 013e849c-2a39-4bc2-83ba-25e6d17682b5 doesn't exist
Execution ended after 0:00:14.525206997
Setting pipeline to NULL ...
2023-07-06T16:15:29.582301Z DEBUG ThreadId(15) tungstenite::protocol: Received close frame: None
2023-07-06T16:15:29.582307Z DEBUG ThreadId(15) tungstenite::protocol: Replying to close with Frame { header: FrameHeader { is_final: true, rsv1: false, rsv2: false, rsv3: false, opcode: Control(Close), mask: None }, payload: [] }
2023-07-06T16:15:29.582314Z INFO ThreadId(15) gst_plugin_webrtc_signalling::server: Received message Ok(Close(None))
2023-07-06T16:15:29.582319Z INFO ThreadId(15) gst_plugin_webrtc_signalling::server: connection closed: None this_id=e4149b2f-30f1-4e55-8452-3c8e82b9255f
2023-07-06T16:15:29.582329Z DEBUG ThreadId(15) remove_peer{peer_id="e4149b2f-30f1-4e55-8452-3c8e82b9255f"}: gst_plugin_webrtc_signalling::server: new
2023-07-06T16:15:29.582342Z DEBUG ThreadId(15) remove_peer{peer_id="e4149b2f-30f1-4e55-8452-3c8e82b9255f"}: gst_plugin_webrtc_signalling::server: close time.busy=5.73µs time.idle=7.54µs
2023-07-06T16:15:29.582355Z DEBUG ThreadId(05) remove_peer{peer_id="e4149b2f-30f1-4e55-8452-3c8e82b9255f"}: gst_plugin_webrtc_signalling::handlers: new
2023-07-06T16:15:29.582364Z INFO ThreadId(05) remove_peer{peer_id="e4149b2f-30f1-4e55-8452-3c8e82b9255f"}: gst_plugin_webrtc_signalling::handlers: removing peer peer_id=e4149b2f-30f1-4e55-8452-3c8e82b9255f
2023-07-06T16:15:29.582376Z DEBUG ThreadId(05) remove_peer{peer_id="e4149b2f-30f1-4e55-8452-3c8e82b9255f"}: gst_plugin_webrtc_signalling::handlers: close time.busy=11.8µs time.idle=9.64µs
Freeing pipeline ...