Skip to content

Fix glimagesink cannot resize viewport when video size changed in caps

Chao Guo requested to merge ChaoGuo/gstreamer:fix_glimagesink_cannot_reszie_viewport_when_caps_video_size_changed into main

glimagesink set preferred viewport size based on the video size in caps that is set in first caps negotiation. If caps renegotiation and video size in caps changes, glimagesink will not resize the viewport according to the new video size, which may cause some problems.

For example, libcamerasrc + glvideomixer:

gst-launch-1.0 libcamerasrc camera-name="device_name" ! video/x-raw,format=YUY2 ! queue ! glupload ! glcolorconvert ! gloverlaycompositor ! glvideomixerelement ! glimagesinkelement

glvideomixer can successfully query latency before libcamerasrc sets caps, and then sets caps first. At this time, the video size in the caps set by glvideomixer is 1X1. When glimagesink receives the event of this caps, it sets the preferred viewport size to 1X1.

When the pipeline sets the real caps, glimagesink will not resize the viewport according to the new caps, resulting in no image display (because the viewport size is 1X1).

0:00:00.290540886 [33m  716[00m 0xffff94000b70 [36mINFO   [00m [00m     videoaggregator gstvideoaggregator.c:1318:gst_video_aggregator_default_negotiated_src_caps:<glvideomixer0:src>[00m set src caps: video/x-raw(memory:GLMemory), format=(string)RGBA, width=(int)1, height=(int)1, framerate=(fraction)25/1, texture-target=(string)2D, pixel-aspect-ratio=(fraction)1/1

... ...

0:00:00.364850139 [33m  716[00m 0xffff94000b70 [37mTRACE  [00m [00m         glimagesink gstglimagesink.c:1912:gst_glimage_sink_show_frame:[00m redisplay texture:1 of size:1x1, window size:1x1
0:00:00.378388182 [33m  716[00m 0xaaaacb9a04a0 [37mDEBUG  [00m [00m         glimagesink gstglimagesink.c:2285:gst_glimage_sink_on_resize:<glimagesink0>[00m GL Window resized to 1x1
0:00:00.378468392 [33m  716[00m 0xaaaacb9a04a0 [37mDEBUG  [00m [00m         glimagesink gstglimagesink.c:2351:gst_glimage_sink_on_resize:<glimagesink0>[00m GL output area now 0,0 1x1
0:00:00.378745107 [33m  716[00m 0xffff94000b70 [37mDEBUG  [00m [00m         glimagesink gstglimagesink.c:2620:gst_glimage_sink_redisplay:[00m Recreating output after mode/size change
0:00:00.378796650 [33m  716[00m 0xffff94000b70 [37mTRACE  [00m [00m         glimagesink gstglimagesink.c:1497:configure_display_from_info:[00m PAR: 1/1 DAR:1/1
0:00:00.378815651 [33m  716[00m 0xffff94000b70 [37mDEBUG  [00m [00m         glimagesink gstglimagesink.c:1501:configure_display_from_info:[00m keeping video height
0:00:00.378835901 [33m  716[00m 0xffff94000b70 [37mDEBUG  [00m [00m         glimagesink gstglimagesink.c:1518:configure_display_from_info:[00m scaling to 1x1
0:00:00.379062157 [33m  716[00m 0xaaaacb9a04a0 [37mDEBUG  [00m [00m         glimagesink gstglimagesink.c:2285:gst_glimage_sink_on_resize:<glimagesink0>[00m GL Window resized to 1x1
0:00:00.379108158 [33m  716[00m 0xaaaacb9a04a0 [37mDEBUG  [00m [00m         glimagesink gstglimagesink.c:2351:gst_glimage_sink_on_resize:<glimagesink0>[00m GL output area now 0,0 1x1

... ...

0:00:00.387645327 [33m  716[00m 0xffff94000b70 [36mINFO   [00m [00m     videoaggregator gstvideoaggregator.c:1318:gst_video_aggregator_default_negotiated_src_caps:<glvideomixer0:src>[00m set src caps: video/x-raw(memory:GLMemory), format=(string)RGBA, width=(int)1920, height=(int)1282, framerate=(fraction)30/1, texture-target=(string)2D, pixel-aspect-ratio=(fraction)1/1

... ...

0:00:00.405695315 [33m  716[00m 0xffff94000b70 [37mTRACE  [00m [00m         glimagesink gstglimagesink.c:1912:gst_glimage_sink_show_frame:[00m redisplay texture:4 of size:1920x1282, window size:1920x1282
0:00:00.406027865 [33m  716[00m 0xaaaacb9a04a0 [37mDEBUG  [00m [00m         glimagesink gstglimagesink.c:2285:gst_glimage_sink_on_resize:<glimagesink0>[00m GL Window resized to 1x1
0:00:00.406077074 [33m  716[00m 0xaaaacb9a04a0 [37mDEBUG  [00m [00m         glimagesink gstglimagesink.c:2351:gst_glimage_sink_on_resize:<glimagesink0>[00m GL output area now 0,0 1x0
0:00:00.406111575 [33m  716[00m 0xaaaacb9a04a0 [37mTRACE  [00m [00m         glimagesink gstglimagesink.c:2404:gst_glimage_sink_on_draw:[00m redrawing texture:4
0:00:00.406158035 [33m  716[00m 0xaaaacb9a04a0 [37mDEBUG  [00m [00m         glimagesink gstglimagesink.c:2417:gst_glimage_sink_on_draw:<glimagesink0>[00m GL output area now 0,0 1x0

So, when glimagesink updates the output format, we need to check if the video size has changed. If so, we need to reset the preferred viewport size.

Merge request reports