eventfd leacks caused by clients disconnected without TEARDOWN request
I am debugging a server, which listens for rtp streams and wraps them arround with rtsp connection. When clients request stream url's from server and disconnect from it without anouncing TEARDOWN request i see that the server doesn't return fome event file descriptors to system.
Scenario
- Server started (0 clients connected)
# lsof -p `ps ax | grep rtp-to-rtsp | grep build | awk '{print $1}'` | wc -l
160
- Client 1 connected to stream (1 client listens for stream)
# lsof -p `ps ax | grep rtp-to-rtsp | grep build | awk '{print $1}'` | wc -l
238
- Client 2 connected to stream (2 clients listen for stream)
# lsof -p `ps ax | grep rtp-to-rtsp | grep build | awk '{print $1}'` | wc -l
304
- Client 2 disconnected from stream
# lsof -p `ps ax | grep rtp-to-rtsp | grep build | awk '{print $1}'` | wc -l
238
- Client 1 disconnected from stream
# lsof -p `ps ax | grep rtp-to-rtsp | grep build | awk '{print $1}'` | wc -l
172
- Client 1 connected to stream
# lsof -p `ps ax | grep rtp-to-rtsp | grep build | awk '{print $1}'` | wc -l
239
- Client 1 disconnected from stream
# lsof -p `ps ax | grep rtp-to-rtsp | grep build | awk '{print $1}'` | wc -l
173
- Client 1 connected to server
- Client 1 disconnected from server
# lsof -p `ps ax | grep rtp-to-rtsp | grep build | awk '{print $1}'` | wc -l
174
Investigation
The clients terminate TCP connection but doesn't send TEARDOWN request. I have attached two logs with opened files, that server holds
log1 - list of opened files by server after first client (connect disconnect)
log2 - list of opened files by server after 10 times connect/disconnect
The diff
between the files shows, that some eventfd's are not freed:
diff -Npru /tmp/log1 /tmp/log2
--- /tmp/log1 2021-06-15 14:11:40.915764953 +0200
+++ /tmp/log2 2021-06-15 14:12:19.296715697 +0200
@@ -168,5 +168,15 @@ rtp-to-rt 26946 ifolbort 39u a_inode
rtp-to-rt 26946 ifolbort 40u IPv4 8903123 0t0 TCP *:8554 (LISTEN)
rtp-to-rt 26946 ifolbort 41u IPv4 8903125 0t0 TCP localhost:3737 (LISTEN)
rtp-to-rt 26946 ifolbort 43u a_inode 0,13 0 9299 [eventfd]
+rtp-to-rt 26946 ifolbort 44u a_inode 0,13 0 9299 [eventfd]
+rtp-to-rt 26946 ifolbort 45u a_inode 0,13 0 9299 [eventfd]
+rtp-to-rt 26946 ifolbort 46u a_inode 0,13 0 9299 [eventfd]
+rtp-to-rt 26946 ifolbort 47u a_inode 0,13 0 9299 [eventfd]
+rtp-to-rt 26946 ifolbort 48u a_inode 0,13 0 9299 [eventfd]
+rtp-to-rt 26946 ifolbort 49u a_inode 0,13 0 9299 [eventfd]
+rtp-to-rt 26946 ifolbort 50u a_inode 0,13 0 9299 [eventfd]
+rtp-to-rt 26946 ifolbort 51u a_inode 0,13 0 9299 [eventfd]
rtp-to-rt 26946 ifolbort 52u unix 0x00000000d7ef03c5 0t0 8904713 type=STREAM
rtp-to-rt 26946 ifolbort 53u unix 0x000000000ecc3ba3 0t0 8904714 type=STREAM
+rtp-to-rt 26946 ifolbort 54u a_inode 0,13 0 9299 [eventfd]
+rtp-to-rt 26946 ifolbort 55u a_inode 0,13 0 9299 [eventfd]
If at least one client is connected to server then the leack is not observed (see steps 3,4 in the Scenario).
I wonder if it is an internal gst-rtsp-server issue or a misshandle my rtsp connections?
For each media factory i use next configurations:
factory = gst_rtsp_media_factory_new();
gst_rtsp_media_factory_set_media_gtype(factory, GST_TYPE_RTSP_MEDIA);
gst_rtsp_media_factory_set_launch(GST_RTSP_MEDIA_FACTORY(factory), gstLaunchString.c_str());
gst_rtsp_media_factory_set_shared(GST_RTSP_MEDIA_FACTORY(factory), FALSE);
gst_rtsp_media_factory_set_suspend_mode(GST_RTSP_MEDIA_FACTORY(factory), GST_RTSP_SUSPEND_MODE_RESET);
gst_rtsp_media_factory_set_transport_mode(GST_RTSP_MEDIA_FACTORY(factory), GST_RTSP_TRANSPORT_MODE_PLAY);
gst_rtsp_media_factory_set_stop_on_disconnect(GST_RTSP_MEDIA_FACTORY(factory), TRUE);
Up on GstRTSPClient
closed
signal i perform next cleanup:
static gboolean cleanup_sessions(GstRTSPServer* server)
{
GstRTSPSessionPool* pool = gst_rtsp_server_get_session_pool(server);
guint sessions = gst_rtsp_session_pool_get_n_sessions(pool);
gst_rtsp_session_pool_cleanup(pool);
gst_object_unref(pool);
return TRUE;
}
GstRTSPFilterResult media_remove(GstRTSPSession * sess,
GstRTSPSessionMedia * media,
gpointer user_data)
{
return GST_RTSP_FILTER_REMOVE;
}
static GstRTSPFilterResult session_remove(GstRTSPClient * client,
GstRTSPSession * sess,
gpointer user_data)
{
gst_rtsp_session_filter(sess, media_remove, nullptr);
return GST_RTSP_FILTER_REMOVE;
}
static void closed_callback (GstRTSPClient * self,
gpointer user_data)
{
GstRTSPConnection* connection = gst_rtsp_client_get_connection(self);
const std::string ipAddr = gst_rtsp_connection_get_ip(connection);
std::cerr << "client (" << ipAddr << ") closed" << std::endl;
gst_rtsp_client_session_filter(self, session_remove, nullptr);
gst_rtsp_client_close(self);
}
static void client_connected (GstRTSPServer * server, GstRTSPClient * client)
{
...
g_signal_connect(client, "closed", G_CALLBACK(closed_callback), nullptr);
...
}
void RtpToRtspserver::init()
{
//create rtsp server. By default it will listen connections on port 8554
mRtspserver = gst_rtsp_server_new();
//Set server signals and callbacks
g_signal_connect(mRtspserver, "client-connected", G_CALLBACK(client_connected), nullptr);
//set up monitoring callbacks
g_timeout_add_seconds(2, G_SOURCE_FUNC(cleanup_sessions), mRtspserver);
...
}