Skip to content

Proposal: Improve Support for Text and Vector Graphics Overlays

With this MR we are going to need better support for efficient rendering of complex text and shape overlays in GStreamer.

Here is a brief overview of the current state of vector overlays:

  • rsvgoverlay (-bad) - render SVG directly into video frame (uses RSVG and Cairo, no OverlayCompositionMeta support)
  • faceoverlay (-bad) - render SVG graphic over detected face in video frame (uses rsvgoverlay)
  • textoverlay (-base) - render a single piece of text (uses Cairo, OverlayCompositionMeta support)
  • clockoverlay, timeoverlay (-base) - uses Cairo

I am proposing:

  1. new GstSvGMeta meta holding a single string, storing an SVG XML string. We will use SVG as our specification for describing text and vector overlays.

Advantage: we don't have to invent a new format, and we already have SVG support. Disadvantage : XML :)

  1. Add OverlayCompositionMeta support to rsvgoverlay

  2. Add new gstskiaoverlay element to make use of Google's skia library. It's a pain in the ass to build, and has little to no documentation, but can be faster than Cairo (need to profile this). skia has experimental support for SVG.

With #1 and #2 (closed), the new mloverlay element can parse meta data from a neural network output, and attach a generated GstSvgMeta and a OverlayCompositionMeta to each video buffer; downstream element will render it. #3 (closed) would be nice to have.

Edited by Aaron Boxer