Skip to content
GitLab
Open
Issue created by Antonio Ospite@ao2

hlsdemux: enable support for external subtitles

HLS streams can have external subtitles (for example using the WebVTT format), the following test streams show the functionality when opened in VLC.

GStreamer does not support this yet: subtitles tracks are ignored and cannot be conditionally selected during operation.

This can be reproduced with:

GST_DEBUG=hlsdemux:6 gst-play-1.0 https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/bipbop_16x9_variant.m3u8

Or with totem.

The logs show some hints:

...
0:00:03.424664491 10087 0x558cc9cd7800 LOG                 hlsdemux gsthlsdemux.c:541:gst_hls_demux_setup_streams:<hlsdemux0> media of type 2 - English, uri: https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/subtitles/eng/prog_index.m3u8
0:00:03.424687226 10087 0x558cc9cd7800 LOG                 hlsdemux gsthlsdemux.c:492:create_stream_for_playlist:<hlsdemux0> Ignoring not-selected stream
0:00:03.424707869 10087 0x558cc9cd7800 LOG                 hlsdemux gsthlsdemux.c:541:gst_hls_demux_setup_streams:<hlsdemux0> media of type 2 - English (Forced), uri: https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/subtitles/eng_forced/prog_index.m3u8
0:00:03.424724994 10087 0x558cc9cd7800 LOG                 hlsdemux gsthlsdemux.c:492:create_stream_for_playlist:<hlsdemux0> Ignoring not-selected stream
0:00:03.424748322 10087 0x558cc9cd7800 LOG                 hlsdemux gsthlsdemux.c:541:gst_hls_demux_setup_streams:<hlsdemux0> media of type 2 - Français, uri: https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/subtitles/fra/prog_index.m3u8
0:00:03.424771185 10087 0x558cc9cd7800 LOG                 hlsdemux gsthlsdemux.c:492:create_stream_for_playlist:<hlsdemux0> Ignoring not-selected stream
0:00:03.424802541 10087 0x558cc9cd7800 LOG                 hlsdemux gsthlsdemux.c:541:gst_hls_demux_setup_streams:<hlsdemux0> media of type 2 - Français (Forced), uri: https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/subtitles/fra_forced/prog_index.m3u8
0:00:03.424822417 10087 0x558cc9cd7800 LOG                 hlsdemux gsthlsdemux.c:492:create_stream_for_playlist:<hlsdemux0> Ignoring not-selected stream
0:00:03.424852221 10087 0x558cc9cd7800 LOG                 hlsdemux gsthlsdemux.c:541:gst_hls_demux_setup_streams:<hlsdemux0> media of type 2 - Español, uri: https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/subtitles/spa/prog_index.m3u8
0:00:03.424873442 10087 0x558cc9cd7800 LOG                 hlsdemux gsthlsdemux.c:492:create_stream_for_playlist:<hlsdemux0> Ignoring not-selected stream
0:00:03.424900684 10087 0x558cc9cd7800 LOG                 hlsdemux gsthlsdemux.c:541:gst_hls_demux_setup_streams:<hlsdemux0> media of type 2 - Español (Forced), uri: https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/subtitles/spa_forced/prog_index.m3u8
0:00:03.424918554 10087 0x558cc9cd7800 LOG                 hlsdemux gsthlsdemux.c:492:create_stream_for_playlist:<hlsdemux0> Ignoring not-selected stream
0:00:03.424943210 10087 0x558cc9cd7800 LOG                 hlsdemux gsthlsdemux.c:541:gst_hls_demux_setup_streams:<hlsdemux0> media of type 2 - 日本語, uri: https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/subtitles/jpn/prog_index.m3u8
0:00:03.424961423 10087 0x558cc9cd7800 LOG                 hlsdemux gsthlsdemux.c:492:create_stream_for_playlist:<hlsdemux0> Ignoring not-selected stream
0:00:03.424986910 10087 0x558cc9cd7800 LOG                 hlsdemux gsthlsdemux.c:541:gst_hls_demux_setup_streams:<hlsdemux0> media of type 2 - 日本語 (Forced), uri: https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/subtitles/jpn_forced/prog_index.m3u8
0:00:03.425006284 10087 0x558cc9cd7800 LOG                 hlsdemux gsthlsdemux.c:492:create_stream_for_playlist:<hlsdemux0> Ignoring not-selected stream
0:00:03.425189806 10087 0x7f25344ecad0 DE
...

With the following quick hack we can at least create auto-selected subtitles tracks, and the default one is even rendered:

diff --git a/ext/hls/gsthlsdemux.c b/ext/hls/gsthlsdemux.c
index 4317d65c3..500d1e8e8 100644
--- a/ext/hls/gsthlsdemux.c
+++ b/ext/hls/gsthlsdemux.c
@@ -541,7 +541,8 @@ gst_hls_demux_setup_streams (GstAdaptiveDemux * demux)
           media->name, media->uri);
       create_stream_for_playlist (demux, media->playlist, FALSE,
           (media->mtype == GST_HLS_MEDIA_TYPE_VIDEO ||
-              media->mtype == GST_HLS_MEDIA_TYPE_AUDIO));
+              media->mtype == GST_HLS_MEDIA_TYPE_AUDIO ||
+              (media->mtype == GST_HLS_MEDIA_TYPE_SUBTITLES && media->autoselect)));

       mlist = mlist->next;
     }

However more work is required for proper support:

  1. changing subtitles does not work, and trying to change them disables them for the rest of the playback, gst-play-1.0 tells me: 
    Switching to subtitle track 2 of 4. 
0:00:18.966069361 15464 0x561dc35f8990 ERROR                  queue gstqueue.c:1028:gst_queue_handle_sink_event:<subqueue> Failed to push event
0:00:18.966229763 15464 0x561dc35f8990 ERROR                  queue gstqueue.c:1028:gst_queue_handle_sink_event:<subqueue> Failed to push event
  2. the track NAME is not shown in totem in the subtitle selection menu (not sure if this is a totem issue).

Ideally, the AUTOSELECT, DEFAULT, and FORCE attributes specified in RFC8216 would be respected as well.

Some of the issue are probably related to the FIXME in create_stream_for_playlist(): https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/blob/1.16.0/ext/hls/gsthlsdemux.c#L491

Thanks, Antonio