[gl] glcolorscale passthrough mode causing playback failure on video resolution change
Submitted by Samuel Hurst
Created attachment 356793
Excerpt of log for the issue described
I've been using GStreamer on a Raspberry Pi 2 to display fullscreen DASH. There is no X session involved, it's using OpenGL ES directly on the framebuffer. As gst-launch-1.0 starts with a window size equal to the video resolution of the first DASH representation received and not fullscreen, I use a custom pipeline to resize the video window to match the output screen size. My pipeline is below:
gst-launch-1.0 souphttpsrc location=$MPD_URL ! dashdemux name=d \
d. ! video/quicktime ! queue ! qtdemux ! h264parse ! omxh264dec ! queue \
! glupload ! glcolorconvert ! glcolorscale \
! "video/x-raw(memory:GLMemory),width=$X_RES,height=$Y_RES" ! queue \
d. ! audio/x-m4a ! queue ! qtdemux ! aacparse ! avdec_aac ! audioconvert \
Where $MPD_URL is the MPD URL of a given stream, and $X_RES and $Y_RES representing the screen size to display on.
This works fine, unless the MPD contains a video representation with the same resolution as the screen output. Then, GStreamer has issues when the resolution changes away from that specified in the capsfilter downstream of the glcolorscale element.
- Player starts, output resolution is 1280x720
- dashdemux selects first representation of a much smaller video resolution than the output resolution
- Playback begins
- Good network conditions mean dashdemux picks a representation with video resolution equal to that of the output resolution of 1280x720
- Playback continues as normal
- Network conditions dictate that dashdemux changes representation again to a representation with a different video resolution to the output resolution
- The pipeline ends with a "not-negotiated" error
I've attached an excerpt of a debug log made with GST_DEBUG="*:3,glfilter:9,basetransform:9,GST_PADS:6" which shows the error (glcolorscale-passthrough.log.xz). Of particular note seem to be the lines which state:
0:00:19.704991143 317 0x25d830 WARN basetransform gstbasetransform.c:1346:gst_base_transform_setcaps:
<gluploadelement0> transform could not transform video/x-raw(memory:GLMemory), format=(string)RGBA, width=(int)960, height=(int)540, interlace-mode=(string)progressive, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)sRGB, framerate=(fraction)50/1 in anything we support
After a bit of investigation, I discovered that it seemed to be that the element was stuck in passthrough mode, and that this was causing the playback to fail because once in passthrough mode the filter could not be brought out of it should the caps change upstream of the element. By removing the check for gst_base_transform_is_passthrough() in gst_gl_filter_transform_caps() in gst-plugins-bad/gst-libs/gst/gl/gstglfilter.c, it now forces it to do a full evaluation of the caps when it changes, instead of just taking a reference to the sink pad caps as they arrive.
By removing the aforementioned check for ::is_passthrough(), this has fixed the issue for me. I will also attach a patch that I'm now using internally. This has fixed the issue for me, but as I'm not an expert in either OpenGL nor the GstBaseTransform side of things I would like to ask what the GStreamer community's take on this is. I can't find any other GstBaseTransform element doing what GstGlFilter is doing with that ::is_passthrough() call, so I thought it a tad suspect.
If you need a test stream, I have created a standalone file which replicates the behaviour of dashdemux which I can provide. Unfortunately, the video is too large to upload here (9.1MB and the file size limit is 3.6MB), but I'm looking into other ways of getting it to you.
For reference, this is running from Git Master as of about three weeks ago with a few patches pulled in for gl and omx improvements ontop of that, and was also reproducible on the 1.12.1 release.
Attachment 356793, "Excerpt of log for the issue described":