Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • gst-plugins-good gst-plugins-good
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 646
    • Issues 646
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 71
    • Merge requests 71
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Container Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • GStreamerGStreamer
  • gst-plugins-goodgst-plugins-good
  • Issues
  • #308
Closed
Open
Issue created Oct 11, 2016 by Bugzilla Migration User@bugzilla-migration

V4L2 compositor proposal

Submitted by Sebastien MATZ

Link to original bug (#772766)

Description

Created attachment 337446
V4L2 compositor code

Hi folks,

Here is a proposal of a new GStreamer element which performs
video composition using video4linux2 M2M devices.

Today we are able to use our new element with simple pipelines using
gst-launch and also more complex ones using our application on our
platform (imx6 board). In both cases, fbdevsink element (with sometimes
others sinks) is used to display the composition's result. Used capture/output
resolution is 720p or 1080p and we use various video format: YUV2, NV12 and RGB16. Ouput IO-mode is always dmabuf-import and capture IO-mode is always dmabuf.

The current status of our development is work-in-progress and well
there is still some bugs. It seems to work as expected with
gst-launch but we experiment different errors when using it in
our application after some start/stop cycles (== gstreamer pipeline reinstanciation from a same process). That is why life cycle of our buffers is greatly suspected. Typically encountered errors are after a failing start are:

ERROR v4l2allocator gstv4l2allocator.c:1245:gst_v4l2_allocator_qbuf:ipu1-capture:pool:src:allocator failed queueing buffer 0: Invalid argument
ERROR v4l2bufferpool gstv4l2bufferpool.c:1127:gst_v4l2_buffer_pool_qbuf:ipu1-capture:pool:src could not queue a buffer 0
ERROR v4l2allocator gstv4l2allocator.c:1245:gst_v4l2_allocator_qbuf:ipu1-capture:pool:src:allocator failed queueing buffer 1: Invalid argument
ERROR v4l2bufferpool gstv4l2bufferpool.c:1127:gst_v4l2_buffer_pool_qbuf:ipu1-capture:pool:src could not queue a buffer 1
(...)

or

ERROR gst_v4l2_compositor_queue_jobs() failed

which means buffers are still owned somewhere in the pipeline (downstream).

or

process freezing :(

A this point I cannot find where is/are the problem(s). We also use
non-standard elements in our application or plaform (gstreamer elements,
drivers, ...). So at this point some help/feedback should be appreciated.
Can someone try to run and compile it on another platform ?
Is there an easy way to restart the pipeline from gst-launch (gstreamer pipeline
reinstanciation from a same process ?)

I put two patches in attachment. Both should apply on the current master branch.
First one (v4l2compositor.patch) is the code of the compositor itself. Note that we use gstvideoaggregator (from gst-plugins-bad) as base class after importing and renaming it in the gst-plugins-good repository.
Second one (testing_with_gst-launch.patch) is only for testing purpose only. It makes some modifications in v4l2convert that we put in front of fbdevsink in some of our gst-launch commands (see below).

Testing v4l2compositor with gst-launch

  
Here is a complete example with gst-launch with some logs. It composes two stream into one. Here all streams are 720p:  
  
  
# GST_DEBUG_NO_COLOR=1 GST_DEBUG=*:3 gst-launch-1.0 -v  v4l2compositor num_jobs=12 device=/dev/v4l/by-path/ipu1-scaler output-io-mode=5  capture-io-mode=4 name=comp sink_0::xpos=8 sink_0::ypos=8 sink_0::width=624 sin k_0::height=344 sink_1::xpos=648 sink_1::ypos=8 sink_1::width=624 sink_1::height =344   ! "video/x-raw,pixel-aspect-ratio=1/1,colorimetry=bt709,height=720,framer ate=24/1,width=1280" ! v4l2convert device=/dev/v4l/by-path/ipu2-scaler output-io-mode=5 capture-io-mode=auto ! fbdevsink v4l2src  device=/dev/v4l/by-path/ipu1-capture io-mode=4 ! "video/x-raw,pixel-aspect-ratio=1/1,colorimetry=bt709,height=720,framerate=24/1,width=1280,format=YUY2" ! comp. v4l2src  device=/dev/v4l/by-path/ipu2-capture io-mode=4 ! "video/x-raw,pixel-aspect-ratio=1/1,colorimetry=bt709,height=720,framerate=24/1,width=1280,format=YUY2" ! comp.  
Setting pipeline to PAUSED ...  
WARN                    v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:`<v4l2transform0>` Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument  
WARN                    v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:`<v4l2transform0>` Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument  
WARN                    v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:`<v4l2transform0>` Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument  
WARN                    v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:`<v4l2transform0>` Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument  
WARN                    v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:`<v4l2transform0>` Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument  
WARN                    v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:`<v4l2transform0>` Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument  
WARN                    v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:`<v4l2transform0>` Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument  
WARN                    v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:`<v4l2transform0>` Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument  
WARN                    v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:`<v4l2transform0>` Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument  
WARN                    v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:`<v4l2transform0>` Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument  
WARN                    v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:`<v4l2transform0>` Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument  
WARN                 v4l2src gstv4l2src.c:545:gst_v4l2src_query:`<v4l2src0>` Can't give latency since framerate isn't fixated !  
Pipeline is live and does not need PREROLL ...  
WARN                 v4l2src gstv4l2src.c:545:gst_v4l2src_query:`<v4l2src1>` Can't give latency since framerate isn't fixated !  
WARN         v4l2_aggregator gstv4l2aggregator.c:1409:gst_v4l2_aggregator_query_latency_unlocked:`<comp>` Latency query failed  
Setting pipeline to PLAYING ...  
New clock: GstSystemClock  
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, framerate=(fraction)24/1  
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, framerate=(fraction)24/1  
/GstPipeline:pipeline0/GstV4l2Compositor:comp.GstV4l2CompositorPad:sink_0: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, framerate=(fraction)24/1  
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, framerate=(fraction)24/1  
/GstPipeline:pipeline0/GstV4l2Src:v4l2src1.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, framerate=(fraction)24/1  
/GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, framerate=(fraction)24/1  
/GstPipeline:pipeline0/GstV4l2Compositor:comp.GstV4l2CompositorPad:sink_1: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, framerate=(fraction)24/1  
/GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, framerate=(fraction)24/1  
WARN          v4l2bufferpool gstv4l2bufferpool.c:748:gst_v4l2_buffer_pool_start:<v4l2src0:pool:src> Uncertain or not enough buffers, enabling copy threshold  
WARN          v4l2bufferpool gstv4l2bufferpool.c:748:gst_v4l2_buffer_pool_start:<v4l2src1:pool:src> Uncertain or not enough buffers, enabling copy threshold  
FIXME               basesink gstbasesink.c:3124:gst_base_sink_default_event:`<fbdevsink0>` stream-start event without group-id. Consider implementing group-id handling in the upstream elements  
WARN               structure gststructure.c:1935:priv_gst_structure_append_to_gstring: No value transform to serialize field 'chroma-site' of type 'string'  
/GstPipeline:pipeline0/GstV4l2Compositor:comp.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive, chroma-site=(string)NULL  
WARN               structure gststructure.c:1935:priv_gst_structure_append_to_gstring: No value transform to serialize field 'chroma-site' of type 'string'  
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive, chroma-site=(string)NULL  
/GstPipeline:pipeline0/GstV4l2Transform:v4l2transform0.GstPad:src: caps = video/x-raw, format=(string)RGB16, width=(int)1280, height=(int)720, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive  
/GstPipeline:pipeline0/GstFBDEVSink:fbdevsink0.GstPad:sink: caps = video/x-raw, format=(string)RGB16, width=(int)1280, height=(int)720, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive  
WARN          v4l2bufferpool gstv4l2bufferpool.c:748:gst_v4l2_buffer_pool_start:<v4l2transform0:pool:src> Uncertain or not enough buffers, enabling copy threshold  
WARN               structure gststructure.c:1935:priv_gst_structure_append_to_gstring: No value transform to serialize field 'chroma-site' of type 'string'  
/GstPipeline:pipeline0/GstV4l2Transform:v4l2transform0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive, chroma-site=(string)NULL  
WARN               structure gststructure.c:1935:priv_gst_structure_append_to_gstring: No value transform to serialize field 'chroma-site' of type 'string'  
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive, chroma-site=(string)NULL  
Redistribute latency...  
  
<Ctrl+C>  
  
Execution ended after 0:00:36.615017671  
Setting pipeline to PAUSED ...  
Setting pipeline to READY ...  
Setting pipeline to NULL ...  
Freeing pipeline ...  
  
  
Regards,  
  
Sebastien  
    
**Patch 337446**, "V4L2 compositor code":    
[v4l2compositor.patch](/uploads/f5668d908701d9deba01c8366be59592/v4l2compositor.patch)
Assignee
Assign to
Time tracking