V4L2 compositor proposal
Submitted by Sebastien MATZ
Link to original bug (#772766)
Description
Created attachment 337446
V4L2 compositor code
Hi folks,
Here is a proposal of a new GStreamer element which performs
video composition using video4linux2 M2M devices.
Today we are able to use our new element with simple pipelines using
gst-launch and also more complex ones using our application on our
platform (imx6 board). In both cases, fbdevsink element (with sometimes
others sinks) is used to display the composition's result. Used capture/output
resolution is 720p or 1080p and we use various video format: YUV2, NV12 and RGB16. Ouput IO-mode is always dmabuf-import and capture IO-mode is always dmabuf.
The current status of our development is work-in-progress and well
there is still some bugs. It seems to work as expected with
gst-launch but we experiment different errors when using it in
our application after some start/stop cycles (== gstreamer pipeline reinstanciation from a same process). That is why life cycle of our buffers is greatly suspected. Typically encountered errors are after a failing start are:
ERROR v4l2allocator gstv4l2allocator.c:1245:gst_v4l2_allocator_qbuf:ipu1-capture:pool:src:allocator failed queueing buffer 0: Invalid argument
ERROR v4l2bufferpool gstv4l2bufferpool.c:1127:gst_v4l2_buffer_pool_qbuf:ipu1-capture:pool:src could not queue a buffer 0
ERROR v4l2allocator gstv4l2allocator.c:1245:gst_v4l2_allocator_qbuf:ipu1-capture:pool:src:allocator failed queueing buffer 1: Invalid argument
ERROR v4l2bufferpool gstv4l2bufferpool.c:1127:gst_v4l2_buffer_pool_qbuf:ipu1-capture:pool:src could not queue a buffer 1
(...)
or
ERROR gst_v4l2_compositor_queue_jobs() failed
which means buffers are still owned somewhere in the pipeline (downstream).
or
process freezing :(
A this point I cannot find where is/are the problem(s). We also use
non-standard elements in our application or plaform (gstreamer elements,
drivers, ...). So at this point some help/feedback should be appreciated.
Can someone try to run and compile it on another platform ?
Is there an easy way to restart the pipeline from gst-launch (gstreamer pipeline
reinstanciation from a same process ?)
I put two patches in attachment. Both should apply on the current master branch.
First one (v4l2compositor.patch) is the code of the compositor itself. Note that we use gstvideoaggregator (from gst-plugins-bad) as base class after importing and renaming it in the gst-plugins-good repository.
Second one (testing_with_gst-launch.patch) is only for testing purpose only. It makes some modifications in v4l2convert that we put in front of fbdevsink in some of our gst-launch commands (see below).
Testing v4l2compositor with gst-launch
Here is a complete example with gst-launch with some logs. It composes two stream into one. Here all streams are 720p:
# GST_DEBUG_NO_COLOR=1 GST_DEBUG=*:3 gst-launch-1.0 -v v4l2compositor num_jobs=12 device=/dev/v4l/by-path/ipu1-scaler output-io-mode=5 capture-io-mode=4 name=comp sink_0::xpos=8 sink_0::ypos=8 sink_0::width=624 sin k_0::height=344 sink_1::xpos=648 sink_1::ypos=8 sink_1::width=624 sink_1::height =344 ! "video/x-raw,pixel-aspect-ratio=1/1,colorimetry=bt709,height=720,framer ate=24/1,width=1280" ! v4l2convert device=/dev/v4l/by-path/ipu2-scaler output-io-mode=5 capture-io-mode=auto ! fbdevsink v4l2src device=/dev/v4l/by-path/ipu1-capture io-mode=4 ! "video/x-raw,pixel-aspect-ratio=1/1,colorimetry=bt709,height=720,framerate=24/1,width=1280,format=YUY2" ! comp. v4l2src device=/dev/v4l/by-path/ipu2-capture io-mode=4 ! "video/x-raw,pixel-aspect-ratio=1/1,colorimetry=bt709,height=720,framerate=24/1,width=1280,format=YUY2" ! comp.
Setting pipeline to PAUSED ...
WARN v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:`<v4l2transform0>` Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument
WARN v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:`<v4l2transform0>` Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument
WARN v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:`<v4l2transform0>` Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument
WARN v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:`<v4l2transform0>` Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument
WARN v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:`<v4l2transform0>` Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument
WARN v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:`<v4l2transform0>` Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument
WARN v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:`<v4l2transform0>` Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument
WARN v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:`<v4l2transform0>` Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument
WARN v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:`<v4l2transform0>` Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument
WARN v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:`<v4l2transform0>` Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument
WARN v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:`<v4l2transform0>` Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument
WARN v4l2src gstv4l2src.c:545:gst_v4l2src_query:`<v4l2src0>` Can't give latency since framerate isn't fixated !
Pipeline is live and does not need PREROLL ...
WARN v4l2src gstv4l2src.c:545:gst_v4l2src_query:`<v4l2src1>` Can't give latency since framerate isn't fixated !
WARN v4l2_aggregator gstv4l2aggregator.c:1409:gst_v4l2_aggregator_query_latency_unlocked:`<comp>` Latency query failed
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, framerate=(fraction)24/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, framerate=(fraction)24/1
/GstPipeline:pipeline0/GstV4l2Compositor:comp.GstV4l2CompositorPad:sink_0: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, framerate=(fraction)24/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, framerate=(fraction)24/1
/GstPipeline:pipeline0/GstV4l2Src:v4l2src1.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, framerate=(fraction)24/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, framerate=(fraction)24/1
/GstPipeline:pipeline0/GstV4l2Compositor:comp.GstV4l2CompositorPad:sink_1: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, framerate=(fraction)24/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, framerate=(fraction)24/1
WARN v4l2bufferpool gstv4l2bufferpool.c:748:gst_v4l2_buffer_pool_start:<v4l2src0:pool:src> Uncertain or not enough buffers, enabling copy threshold
WARN v4l2bufferpool gstv4l2bufferpool.c:748:gst_v4l2_buffer_pool_start:<v4l2src1:pool:src> Uncertain or not enough buffers, enabling copy threshold
FIXME basesink gstbasesink.c:3124:gst_base_sink_default_event:`<fbdevsink0>` stream-start event without group-id. Consider implementing group-id handling in the upstream elements
WARN structure gststructure.c:1935:priv_gst_structure_append_to_gstring: No value transform to serialize field 'chroma-site' of type 'string'
/GstPipeline:pipeline0/GstV4l2Compositor:comp.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive, chroma-site=(string)NULL
WARN structure gststructure.c:1935:priv_gst_structure_append_to_gstring: No value transform to serialize field 'chroma-site' of type 'string'
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive, chroma-site=(string)NULL
/GstPipeline:pipeline0/GstV4l2Transform:v4l2transform0.GstPad:src: caps = video/x-raw, format=(string)RGB16, width=(int)1280, height=(int)720, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFBDEVSink:fbdevsink0.GstPad:sink: caps = video/x-raw, format=(string)RGB16, width=(int)1280, height=(int)720, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive
WARN v4l2bufferpool gstv4l2bufferpool.c:748:gst_v4l2_buffer_pool_start:<v4l2transform0:pool:src> Uncertain or not enough buffers, enabling copy threshold
WARN structure gststructure.c:1935:priv_gst_structure_append_to_gstring: No value transform to serialize field 'chroma-site' of type 'string'
/GstPipeline:pipeline0/GstV4l2Transform:v4l2transform0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive, chroma-site=(string)NULL
WARN structure gststructure.c:1935:priv_gst_structure_append_to_gstring: No value transform to serialize field 'chroma-site' of type 'string'
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive, chroma-site=(string)NULL
Redistribute latency...
<Ctrl+C>
Execution ended after 0:00:36.615017671
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
Regards,
Sebastien
**Patch 337446**, "V4L2 compositor code":
[v4l2compositor.patch](/uploads/f5668d908701d9deba01c8366be59592/v4l2compositor.patch)