ges-launch-1.0 clipping getting stuck on HLS discontinuity

Hello developers.

Description

I observed an abnormal behaviour with GStreamer’s editing tool ges-launch-1.0 (version 1.16.1) recently that resulted in stuck processes that would not exit.

Our application case simply cuts an input HLS stream (audio+video), that’s the only thing ges-launch-1.0 would do in that scope. It’s all OK but sometimes the HLS we process have a particularly interesting discontinuity. If the HLS goes from stereo fragments (1 AAC stream with 2 channels) to a segment with an AAC stream with 0 channels then it gets stuck.

      (the segments have video too, just showing the audio here)

    a0.ts                 a1.ts           b8.ts            a2.ts
-------------------------------------------------------------------------
|  1 aac stream   |  1 aac stream   |  1 aac stream   |  1 aac stream   |
| with 2 channels | with 2 channels | with 0 channels | with 2 channels |
-------------------------------------------------------------------------
                                    X                 X

                                               X: EXT-X-DISCONTINUITY

It’s clearly the case of a buggy HLS.

But... do you guys know how can we protect against those cases so we don’t end up with various processes stuck forever?

Reproduction Steps

In order to facilitate debugging, we have created a simple manifest and we have manually included only one problematic segment. The behaviour is the same as using the whole problematic original manifest.

You can reproduce the issue by simply running:

$ ges-launch-1.0 +clip https://watchity-ramon-videos-ireland.s3-eu-west-1.amazonaws.com/rabbit/rabbit.m3u8 i=0 d=20 -o out.ogg

And you will see how it gets stuck at second 13.08.

<position: 0:00:12.920000000 duration: 0:00:20.000000000/>
<position: 0:00:13.080000000 duration: 0:00:20.000000000/>
0:00:42.266508848  4139 0x7f5e5c002680 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.160000000 < 0:00:13.620000000)
0:00:42.273261779  4139 0x7f5e5c002680 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.040000000 < 0:00:13.620000000)
0:00:42.278908361  4139 0x7f5e5c002680 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.080000000 < 0:00:13.620000000)
0:00:42.281074149  4139 0x7f5e5c002680 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.120000000 < 0:00:13.620000000)
0:00:42.283356575  4139 0x7f5e5c002680 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.160000000 < 0:00:13.620000000)
0:00:42.285818489  4139 0x7f5e5c002680 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.160000000 < 0:00:13.620000000)
0:00:42.289001944  4139 0x7f5e5c002680 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.200000000 < 0:00:13.620000000)
0:00:42.293169852  4139 0x7f5e5c002680 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.240000000 < 0:00:13.620000000)
<position: 0:00:13.080000000 duration: 0:00:20.000000000/>
0:00:42.378823196  4139 0x7f5e5c002680 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.280000000 < 0:00:13.620000000)
0:00:42.489892422  4139 0x7f5e5c002680 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.320000000 < 0:00:13.620000000)
0:00:42.599481829  4139 0x7f5e5c002680 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.360000000 < 0:00:13.620000000)
0:00:42.706562443  4139 0x7f5e5c002680 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.400000000 < 0:00:13.620000000)
0:00:42.814879764  4139 0x7f5e5c002680 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.440000000 < 0:00:13.620000000)
0:00:42.928247916  4139 0x7f5e5c002680 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.480000000 < 0:00:13.620000000)
0:00:43.033689309  4139 0x7f5e5c002680 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.520000000 < 0:00:13.620000000)
<position: 0:00:13.080000000 duration: 0:00:20.000000000/>
0:00:43.270556967  4139 0x7f5e5c002680 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.600000000 < 0:00:13.620000000)
<position: 0:00:13.080000000 duration: 0:00:20.000000000/>
<position: 0:00:13.080000000 duration: 0:00:20.000000000/>
<position: 0:00:13.080000000 duration: 0:00:20.000000000/>
<position: 0:00:13.080000000 duration: 0:00:20.000000000/>
<position: 0:00:13.080000000 duration: 0:00:20.000000000/>
...

If you instead play it (by omitting the -o flag) then you will see it plays the entire video but doesn’t close the window afterwards as it does with healthy HLS streams.

<position: 0:00:05.164808141 duration: 0:00:20.000000001/>
0:00:13.274872869  4088 0x7fd1700019e0 WARN   	codecparsers_h264 gsth264parser.c:1237:gst_h264_parser_parse_sei_message: Bit non equal to one.
0:00:13.274895193  4088 0x7fd1700019e0 WARN   	codecparsers_h264 gsth264parser.c:1243:gst_h264_parser_parse_sei_message: Bit non equal to zero.
0:00:13.274900599  4088 0x7fd1700019e0 WARN   	codecparsers_h264 gsth264parser.c:1243:gst_h264_parser_parse_sei_message: Bit non equal to zero.
0:00:13.677449587  4088 0x7fd1700019e0 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.160000000 < 0:00:13.620000000)
0:00:13.685138590  4088 0x7fd1700019e0 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.040000000 < 0:00:13.620000000)
0:00:13.688876097  4088 0x7fd1700019e0 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.080000000 < 0:00:13.620000000)
0:00:13.695164645  4088 0x7fd1700019e0 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.120000000 < 0:00:13.620000000)
0:00:13.699870528  4088 0x7fd1700019e0 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.160000000 < 0:00:13.620000000)
0:00:13.701827595  4088 0x7fd1700019e0 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.160000000 < 0:00:13.620000000)
0:00:13.707254573  4088 0x7fd1700019e0 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.200000000 < 0:00:13.620000000)
0:00:13.714031598  4088 0x7fd1700019e0 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.240000000 < 0:00:13.620000000)
0:00:13.747354620  4088 0x7fd1700019e0 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.280000000 < 0:00:13.620000000)
0:00:13.765969317  4088 0x7fd1700019e0 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.320000000 < 0:00:13.620000000)
0:00:13.800168936  4088 0x7fd1700019e0 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.360000000 < 0:00:13.620000000)
0:00:13.834934563  4088 0x7fd1700019e0 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.400000000 < 0:00:13.620000000)
0:00:13.878851712  4088 0x7fd1700019e0 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.440000000 < 0:00:13.620000000)
0:00:13.932485432  4088 0x7fd1700019e0 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.480000000 < 0:00:13.620000000)
0:00:13.995678540  4088 0x7fd1700019e0 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.520000000 < 0:00:13.620000000)
0:00:14.016629730  4088 0x7fd1700019e0 WARN        	videodecoder gstvideodecoder.c:2762:gst_video_decoder_prepare_finish_frame:<avdec_h264-1> decreasing timestamp (0:00:13.600000000 < 0:00:13.620000000)
0:00:20.073656162  4088   	0x786940 WARN          	aggregator gstaggregator.c:1757:gst_aggregator_query_latency_unlocked:<gessmartmixer0-compositor> Latency query failed
0:00:20.074010609  4088 0x7fd1d489b1e0 WARN          	aggregator gstaggregator.c:1757:gst_aggregator_query_latency_unlocked:<gessmartmixer0-compositor> Latency query failed
0:00:20.102896990  4088 0x7fd1d46a1e80 WARN                 	ges ges-smart-video-mixer.c:189:parse_metadata: The current source should use a framepositioner
<position: 0:00:20.000000000 duration: 0:00:20.000000001/>
<position: 0:00:20.000000000 duration: 0:00:20.000000001/>
<position: 0:00:20.000000000 duration: 0:00:20.000000001/>
<position: 0:00:20.000000000 duration: 0:00:20.000000001/>
<position: 0:00:20.000000000 duration: 0:00:20.000000001/>
<position: 0:00:20.000000000 duration: 0:00:20.000000001/>
<position: 0:00:20.000000000 duration: 0:00:20.000000001/>
...

Here’s the manifest (rabbit.m3u8):

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:8
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:8.400000,
rabbit0.ts
#EXTINF:4.560000,
rabbit1.ts
#EXT-X-DISCONTINUITY
#EXTINF:3.840000,
copy734.ts   <------------- PROBLEMATIC TS,  if you replace it with rabbit2.ts works
#EXT-X-DISCONTINUITY
#EXTINF:5.640000,
rabbit3.ts
#EXTINF:1.880000,
rabbit4.ts
#EXTINF:2.160000,
rabbit5.ts
#EXTINF:3.800000,
rabbit6.ts
#EXT-X-ENDLIST

The problematic segment has an AAC stream with 0 channels in it...

$ ffprobe -hide_banner copy734.ts
[h264 @ 0x7f8339816600] non-existing SPS 0 referenced in buffering period
[h264 @ 0x7f8339816600] SPS unavailable in decode_picture_timing
[h264 @ 0x7f8339816600] non-existing SPS 0 referenced in buffering period
[h264 @ 0x7f8339816600] SPS unavailable in decode_picture_timing
[mpegts @ 0x7f8339802a00] start time for stream 1 is not set in estimate_timings_from_pts
[mpegts @ 0x7f8339802a00] Could not find codec parameters for stream 1 (Audio: aac ([15][0][0][0] / 0x000F), 0 channels, fltp): unspecified sample rate
Consider increasing the value for the 'analyzeduration' and 'probesize' options
Input #0, mpegts, from 'copy734.ts':
  Duration: 00:00:03.84, start: 0.200000, bitrate: 10457 kb/s
  Program 1
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: h264 (Constrained Baseline) ([27][0][0][0] / 0x001B), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:1[0x101]: Audio: aac ([15][0][0][0] / 0x000F), 0 channels, fltp

... whereas the rest of the segments are stereo:

$ ffprobe -hide_banner rabbit0.ts
Input #0, mpegts, from 'rabbit0.ts':
  Duration: 00:00:08.42, start: 1.400000, bitrate: 4126 kb/s
  Program 1
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: h264 (Constrained Baseline) ([27][0][0][0] / 0x001B), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:1[0x101](und): Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 129 kb/s

Observations

GStreamer won’t get stuck If instead of 1 aac stream with 0 channels we simply remove the aac stream entirely resulting in 1 video stream alone, then it’s fine, it won’t get stuck.
GStreamer won’t get stuck if instead of 1 aac stream with 0 channels we give it a mono aac stream (1 channel), this works and doesn’t get stuck.

Admin message

ges-launch-1.0 clipping getting stuck on HLS discontinuity

Description

Reproduction Steps

Observations