dvdspu: figure out how to make it work with hardware decoders and subpicture overlays
Submitted by Jan Schmidt @thaytan
Assigned to Jan Schmidt @thaytan
Link to original bug (#685282)
Description
Getting the DVD SPU to paint generically is a requirement for allowing the DVD elements to plug / output hardware decoder caps.
Here's a conversation we had about it on IRC:
Sep 26 01:37:04 * thaytan wonders how SPU works
Sep 26 01:37:19 <bilboed> thaytan, with vdpau ?
Sep 26 01:37:24 <thaytan> nod
Sep 26 01:37:35 <thaytan> how it should work, that is
Sep 26 01:37:36 <bilboed> they have also VdpBitmapSurface
Sep 26 01:38:01 <bilboed> so VdpVideoSurface => YUV stuff, VdpBitmapSurface => RGB stuff
Sep 26 01:38:11 <bilboed> VdpOutputSurface => output of the compositor for display
Sep 26 01:38:40 <bilboed> you can create VdpBitmapSurface and map(write) stuff on it
Sep 26 01:39:01 <bilboed> then you give it to the compositor with coordinates (and whatever else) and that's basically it
Sep 26 01:39:09 <thaytan> hmmm
Sep 26 01:39:26 <bilboed> so as a result ... I'm also gonna have to figure out how to solve hw-compositing
Sep 26 01:39:28 <thaytan> not sure I see how that fits into a GStreamer/rsndvdbin context
Sep 26 01:39:31 <bilboed> in a generic fashion that is
Sep 26 01:39:55 <bilboed> thaytan, was thinking you could slap GstOverlayMeta (or sth like that) with attached GstBuffers
Sep 26 01:40:06 <bilboed> thaytan, maybe videomixer or some generic element could do that
Sep 26 01:40:09 <thaytan> I guess it's a vdpspu element with video and subpicture inputs as currently with dvdspu
Sep 26 01:40:25 <thaytan> except the video pad accepts vdp output surface caps
Sep 26 01:40:38 <bilboed> I'd prefer to avoid creating yet-another-custom-element
Sep 26 01:40:45 <thaytan> but, I don't know what it outputs
Sep 26 01:41:12 <thaytan> bilboed: I don't know how to build it generically
Sep 26 01:41:48 <thaytan> the dvdspu element uses the video stream passing through to a) paint onto b) uses the timestamps to crank the SPU state machine
Sep 26 01:42:44 <bilboed> what do you need more apart from "put this RGB bitmap at these coordinates for this timestamp and this duration"
Sep 26 01:43:48 <thaytan> it needs the timestamps and segment info on the video stream so it knows which pixels to generate
Sep 26 01:44:04 <thaytan> it's more "here's a video frame, what's the overlay?"
Sep 26 01:44:13 <thaytan> also, dvdspu works in YUV too
Sep 26 01:44:13 <bilboed> sure, but it doesn't care about the content of that frame
Sep 26 01:44:28 <thaytan> bilboed: not if it's not doing the compositing, no
Sep 26 01:44:34 <bilboed> right
Sep 26 01:45:11 <bilboed> so it could see the stream go through, watch/collect segment/timestamps and decide what subpicture to attach to it (without actually doing any processing and letting downstream handle that)
Sep 26 01:45:13 <thaytan> but the model is still to pass the video buffer stream through the spu element so it can see the timing info it needs, right?
Sep 26 01:45:22 <thaytan> oh, of course
Sep 26 01:45:28 <thaytan> that's what I was suggesting, I guess I wasn't clear
Sep 26 01:45:35 <__tim> thaytan, dvdspu should support GstVideoOverlayComposition imho
Sep 26 01:45:38 <bilboed> I'm not that familiar with SPU
Sep 26 01:45:42 <bilboed> also, what __tim said :)
Sep 26 01:45:54 <bilboed> like that I don't have to solve it in 500 different elements
Sep 26 01:45:56 <thaytan> ok, I guess I'll have to look at the GstVideoOverlayComposition API
Sep 26 01:46:54 * bilboed is not looking forward "at all" to fixing this for cluster-gst
Sep 26 01:47:12 <__tim> it's very dumb, you can provide one or more ARGB or AYUV rectangles and either use helper API to put them onto the raw video, or attach them to the buffer; the sink or whatever can then take over the overlaying using that
Sep 26 01:47:40 <__tim> and it will do a bunch of conversions for you and cache them if the sink or whatever does the overlaying doesn't support what you supplied
Sep 26 01:48:54 <thaytan> well, that sounds feasible - although less efficient than dvdspu painting natively if the composite is in software
Sep 26 01:49:28 <thaytan> maybe it can be extended to add RLE AYUV rectangles as a format though?
Sep 26 01:49:44 <__tim> thaytan, how sure are you of that? because basically you have to parse the RLE data for every single frame, right? is that really so much faster than blitting some ready-made rectangle using orc?
Sep 26 01:50:04 <thaytan> dvdspu gets to skip a lot of transparent pixels
Sep 26 01:52:21 <__tim> yeah, but it's if else and loops etc. You might be right, I'm just curious how much difference it actually makes. Also, you don't have to use the API to blit your pixels, you can still do that as you do now and only attach the AYUV rectangle if downstream supports that
Sep 26 01:54:13 <__tim> it's just convenient because you only have one code path
Sep 26 01:55:01 <thaytan> it sounds like a structural improvement