Basic sharing of video between apps works with the following constraints:
- Format negotiation is needed between nodes. This is fine for simple producer/consumer but is not ideal for generic video filters. We need to have a generic canonical format before we can think about building video filters.
In the work branch it is now possible to let an application provide memory for the buffers. This makes it possible to use avoid some memory copies (a compositor directly rendering into shared DMA-Buf).
We would want something like http://syphon.v002.info/ and for that we probably need to settle for a common video format that is:
- linear RGBA
- HDR capable (possibly float 32 or 16 bits per component)
- Premultiplied alpha
- something that can easily be HW accelerated in the GPU
- fixed framerate
This would allow us to make something similar to the DSP audio path.
There is a vulkan-compute source in the master branch that generates float32 RGBA buffers in DMA-Buf. This seems to work well as a format. More tests need to be done to implement some vulkan shader filters and implement a vulkan swap chain in the SDL renderer.