MAJOR memory leak in rtpjitterbuffer or nvh264sldec (or other element/component)
Title needs update but too few details are known at this time.
Using 1.18.2.
Between running time 127:35:04 and 127:35:10 (approx six seconds) the process working set increases from 128 MB (it has been stable for several days) to 210 MB. It never goes back.
We also notice that no (or very few) frames are beeing rendered during this time period.
The same stream is beeing received and shown by seven other processes on the same machine. They were not affected at this time.
Do anyone have any idea what might be happening?
Is there any suggestions what to do to reproduce or pinpoint this issue? It can take days for the issue to surface, so any idea how to force it happening earlier would be welcome.
Pipeline:
udpsrc auto-multicast=true uri=udp://multicastgroup:port caps="application/x-rtp,clock-rate=90000" ! rtpjitterbuffer latency=200 ! queue ! rtph264depay ! tee ! queue ! h264parse ! nvh264sldec ! overlaycomposition ! d3d11videosink
Logs with log level 2:
127:33:51.501170000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:3300:gst_video_decoder_clip_and_push_buf:<nvh264sldec> Dropping frame due to QoS. start:127:33:15.569675464 deadline:127:33:15.569675464 earliest_time:127:33:15.579599259
127:33:52.491528000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:3300:gst_video_decoder_clip_and_push_buf:<nvh264sldec> Dropping frame due to QoS. start:127:33:16.569607630 deadline:127:33:16.569607630 earliest_time:127:33:16.574895921
127:33:53.685618000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:3300:gst_video_decoder_clip_and_push_buf:<nvh264sldec> Dropping frame due to QoS. start:127:33:17.769621067 deadline:127:33:17.769621067 earliest_time:127:33:17.776554149
127:33:56.093732000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:3300:gst_video_decoder_clip_and_push_buf:<nvh264sldec> Dropping frame due to QoS. start:127:33:20.169876588 deadline:127:33:20.169876588 earliest_time:127:33:20.182848938
127:34:28.495562000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:3300:gst_video_decoder_clip_and_push_buf:<nvh264sldec> Dropping frame due to QoS. start:127:33:52.570424086 deadline:127:33:52.570424086 earliest_time:127:33:52.574280983
127:34:44.093933000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:3300:gst_video_decoder_clip_and_push_buf:<nvh264sldec> Dropping frame due to QoS. start:127:34:08.170356249 deadline:127:34:08.170356249 earliest_time:127:34:08.178877791
127:34:45.298359000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:3300:gst_video_decoder_clip_and_push_buf:<nvh264sldec> Dropping frame due to QoS. start:127:34:09.370298244 deadline:127:34:09.370298244 earliest_time:127:34:09.399008141
127:34:53.298641000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:3300:gst_video_decoder_clip_and_push_buf:<nvh264sldec> Dropping frame due to QoS. start:127:34:17.371050565 deadline:127:34:17.371050565 earliest_time:127:34:17.373429321
127:34:53.697507000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:3300:gst_video_decoder_clip_and_push_buf:<nvh264sldec> Dropping frame due to QoS. start:127:34:17.771041573 deadline:127:34:17.771041573 earliest_time:127:34:17.781352145
127:35:04.482119000 13872 0C64D540 WARN rtpjitterbuffer rtpjitterbuffer.c:576:calculate_skew: delta - skew: 0:00:04.287254151 too big, reset skew
127:35:07.640015000 13872 0C64D540 WARN rtpjitterbuffer rtpjitterbuffer.c:576:calculate_skew: delta - skew: 0:00:01.007024104 too big, reset skew
127:35:07.648470000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:32.858659844 < 127:34:33.825661726)
127:35:07.679249000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:32.898644735 < 127:34:33.825661726)
127:35:07.715200000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:32.938572749 < 127:34:33.825661726)
127:35:07.754737000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:32.978272967 < 127:34:33.825661726)
127:35:07.794253000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:33.017490863 < 127:34:33.825661726)
127:35:07.872185000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:33.055513553 < 127:34:33.825661726)
127:35:07.874816000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:33.092636647 < 127:34:33.825661726)
127:35:07.911296000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:33.128006440 < 127:34:33.825661726)
127:35:07.949954000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:33.161035281 < 127:34:33.825661726)
127:35:07.992787000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:33.189963819 < 127:34:33.825661726)
127:35:08.052016000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:33.215111660 < 127:34:33.825661726)
127:35:08.092175000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:33.237617514 < 127:34:33.825661726)
127:35:08.115284000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:33.255444131 < 127:34:33.825661726)
127:35:08.155444000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:33.268155530 < 127:34:33.825661726)
127:35:08.201872000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:33.275387295 < 127:34:33.825661726)
127:35:08.247490000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:33.276976517 < 127:34:33.825661726)
127:35:09.193605000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:33.291666839 < 127:34:33.825661726)
127:35:09.253468000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:33.331560262 < 127:34:33.825661726)
127:35:09.289414000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:33.371518145 < 127:34:33.825661726)
127:35:09.314811000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:33.411514048 < 127:34:33.825661726)
127:35:09.353151000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:33.451497975 < 127:34:33.825661726)
127:35:09.392705000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:33.491485422 < 127:34:33.825661726)
127:35:09.445572000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:33.531496166 < 127:34:33.825661726)
127:35:09.471869000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:33.571496086 < 127:34:33.825661726)
127:35:09.519229000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:33.611054056 < 127:34:33.825661726)
127:35:09.554555000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:33.650979065 < 127:34:33.825661726)
127:35:09.594360000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:33.690972571 < 127:34:33.825661726)
127:35:09.647677000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:33.730972172 < 127:34:33.825661726)
127:35:09.688890000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:33.770972163 < 127:34:33.825661726)
127:35:09.712108000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:2910:gst_video_decoder_prepare_finish_frame:<nvh264sldec> decreasing timestamp (127:34:33.810972163 < 127:34:33.825661726)
127:35:11.492268000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:3300:gst_video_decoder_clip_and_push_buf:<nvh264sldec> Dropping frame due to QoS. start:127:34:35.571005496 deadline:127:34:35.571005496 earliest_time:127:34:35.590292363
127:35:24.691828000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:3300:gst_video_decoder_clip_and_push_buf:<nvh264sldec> Dropping frame due to QoS. start:127:34:48.771058059 deadline:127:34:48.771058059 earliest_time:127:34:48.774765607
127:35:57.101826000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:3300:gst_video_decoder_clip_and_push_buf:<nvh264sldec> Dropping frame due to QoS. start:127:35:21.171647597 deadline:127:35:21.171647597 earliest_time:127:35:21.202421883
127:36:05.488307000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:3300:gst_video_decoder_clip_and_push_buf:<nvh264sldec> Dropping frame due to QoS. start:127:35:29.571964519 deadline:127:35:29.571964519 earliest_time:127:35:29.575499606
127:36:11.495026000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:3300:gst_video_decoder_clip_and_push_buf:<nvh264sldec> Dropping frame due to QoS. start:127:35:35.571959710 deadline:127:35:35.571959710 earliest_time:127:35:35.599668474
127:36:21.101055000 13872 0C64D6F8 WARN videodecoder gstvideodecoder.c:3300:gst_video_decoder_clip_and_push_buf:<nvh264sldec> Dropping frame due to QoS. start:127:35:45.172180903 deadline:127:35:45.172180903 earliest_time:127:35:45.187507691