Skip to content

Introduce CODEC Alpha plugin

Nicolas Dufresne requested to merge ndufresne/gst-plugins-bad:codec-alpha into master

This plugin contains a set of utility elements allowing to extract, decode and combine CODEC (typically VP8/VP9) alpha stream. This was implemented in order to avoid having massive rework of the decoders internal in order to handle two streams internally. Wrappers for vp8dec and vp9dec are being provided in this MR. Wrappers for HW decoders should also be possible. The idea with keeping per decoder wrapper is to:

  • keep the complexity low
  • make sure sw/hw classification stay clear
  • avoid ranking complexity (we just keep the same rank)
  • Allow for API specific transformation (e.g. v4l2transform, vaapipostproc)

A typical wrapper pipeline would look like:

graph LR
  subgraph codecalphademux
    AD_sink[sink]
    AD_src[src]
    AD_alpha[alpha]
    AD_sink --> AD_src
    AD_sink --> AD_alpha
  end

  subgraph multiqueue0
    MQ_sink0[sink0]
    MQ_src0[src0]
    MQ_sink0 --> MQ_src0

    MQ_sink1[sink1]
    MQ_src1[src1]
    MQ_sink1 --> MQ_src1
  end

  subgraph vp9dec0
    DEC1_sink[sink]
    DEC1_src[src]
    DEC1_sink --> DEC1_src
  end

  subgraph vp9dec1
    DEC2_sink[sink]
    DEC2_src[src]
    DEC2_sink --> DEC2_src
  end

  subgraph alphacombine
    AC_sink[sink]
    AC_alpha[alpha]
    AC_src[src]
    AC_sink --> AC_src
    AC_alpha --> AC_src
  end

  IN_CAPS --> AD_sink

  AD_src --> MQ_sink0
  MQ_src0 --> DEC1_sink
  DEC1_src --> AC_sink

  AD_alpha --> MQ_sink1
  MQ_src1 --> DEC2_sink
  DEC2_src --> AC_alpha

  AC_src --> OUT_CAPS

  IN_CAPS[video/x-vp9,codec-alpha=true]
  OUT_CAPS[video/x-raw,format=A420]

Note: Consider this as an early MR, a lot of documentation work is missing, and HW decoders are neither supported or opt-out yet. As most HW decoders produces NV12, the zero-copy method used in alphacombine would require a new format (e.g. AV12, I/UV/A). Note that this method of combining helps delaying the final transformation so that this transformation happens inside the display sink, making the best / right decision regardless which of decodebin, or decodebin2 you are using.

gst-plugins-base!1128 (merged)

Edited by Nicolas Dufresne

Merge request reports