Skip to content

GitLab

Closed
Created by Eric Yeargan@eric.yeargan

WebRTC DTLS handshake failure: SSL_do_handshake:connection type not set

Platform: Windows 10 Pro 64-bit MSVC

Affected GStreamer versions: 1.15.1, 1.16.0

WebRTC peers: Electron 4.1.1 (Chromium Windows 69.0.3497.128) as well as more recent Chromium versions

When initiating a WebRTC PeerConnection (either offering or answering), the DTLS handshake fails apparently due to this OpenSSL error: "SSL_do_handshake:connection type not set:ssl/ssl_lib.c:3554:"

On the GStreamer side, with *tls* logs set to level 5, I see this sequence repeatedly:

                0:00:11.835289000 15540 00000286D591D440 DEBUG                dtlsdec gstdtlsdec.c:548:sink_chain:<dtlsdec0> received buffer from rtp_0_2873075822 with length 155
                0:00:11.847957000 15540 00000286D591D440 LOG           dtlsconnection gstdtlsconnection.c:629:log_state:<GstDtlsConnection@00000286D56A6890> process start: role=server buf=(00000286D5D3E130:0/155) 10110|0 before SSL initialization
                0:00:11.869501000 15540 00000286D591D440 LOG           dtlsconnection gstdtlsconnection.c:629:log_state:<GstDtlsConnection@00000286D56A6890> process after read: role=server buf=(00000286D5D3E130:0/155) 10110|0 before SSL initialization
                0:00:11.887298000 15540 00000286D591D440 LOG           dtlsconnection gstdtlsconnection.c:629:log_state:<GstDtlsConnection@00000286D56A6890> poll: before handshake: role=server buf=(00000286D5D3E130:0/155) 10110|0 before SSL initialization
                0:00:11.904438000 15540 00000286D591D440 LOG           dtlsconnection gstdtlsconnection.c:629:log_state:<GstDtlsConnection@00000286D56A6890> poll: after handshake: role=server buf=(00000286D5D3E130:0/155) 10110|0 before SSL initialization
                0:00:11.920380000 15540 00000286D591D440 DEBUG         dtlsconnection gstdtlsconnection.c:758:openssl_poll:<GstDtlsConnection@00000286D56A6890> do_handshake encountered BIO error
                0:00:11.935830000 15540 00000286D591D440 ERROR         dtlsconnection gstdtlsconnection.c:771:openssl_poll:<GstDtlsConnection@00000286D56A6890> SSL error
                0:00:11.946996000 15540 00000286D591D440 ERROR         dtlsconnection gstdtlsconnection.c:727:ssl_err_cb:<GstDtlsConnection@00000286D56A6890> ssl error: 26156:error:140B4090:SSL routines:SSL_do_handshake:connection type not set:ssl/ssl_lib.c:3554:

                0:00:11.968253000 15540 00000286D591D440 LOG           dtlsconnection gstdtlsconnection.c:629:log_state:<GstDtlsConnection@00000286D56A6890> process after poll: role=server buf=(00000286D5D3E130:0/155) 10110|0 before SSL initialization
                0:00:11.987237000 15540 00000286D591D440 DEBUG         dtlsconnection gstdtlsconnection.c:551:gst_dtls_connection_process:<GstDtlsConnection@00000286D56A6890> read result: -1

On the Chrome side, webrtc-internals shows that a candidate pair was successfully selected (googReadable/googWritable are true) but the PeerConnection's ICE connection state gets stuck in "checking". The Chrome debug logs show that there's a DTLS timeout.

Based on the "SSL_do_handshake:connection type not set" error, it appears that SSL_set_connect_state()/SSL_set_accept_state() hasn't been called on GstDtlsConnection's OpenSSL handle which should happen in gst_dtls_connection_start() but I never see the corresponding "log_state:initial state set:" message.

GStreamer 1.14.1 doesn't seem to have this issue.

Edited by Eric Yeargan