playbin3: Add a mechanism to avoid requiring apps to connect synchronously to the bus to do stream selection
Right now the correct way to select streams with playbin3/decodebin3/uridecodebin3 is by connecting synchronously on the bus to listen to the STREAM_COLLECTION
signal, and from that callback, send a SELECT_STREAMS
event to the elements. While this work it is not ideal as the idea of having a pipeline bus is to hide threading away from the application, which is not the case here.
This was discussed on IRC and I am pasting here the conversation verbatim:
bilboed: Was the SELECT_STREAM message design to be answered on an async bus
handler ? I observed that it was needed to be done synchronously which it a bit
cumbersome for apps
bilboed message ? There's only an event
slomo you mean the STREAM_COLLECTION message?
and based on that you'd like to do the initial SELECT_STREAM event to get a
SELECTED_STREAMS message with the streams you wanted, without having the
element first select its own default selection? then you need to handle it
from a sync handler indeed
bilboed and yes, ideally it should be answered synchronously. But if you
*already* have preferences (like, you prefer french audio), it's quite fast to
match, no ? you can always preroll first :)
slomo that's rather awkward too though, you'd have to handle all the pads that
appear. select things. pads might disappear, new ones might appear, etc. and
you'd need to remove sinks potentially without deadlocking i think the only way
to handle that cleanly is via a sync message handler for now
bilboed pads might disappear ? Not unless you decide to disable a certain stream
type (like no audio)
thiblahute Right, I meant STREAM_COLLECTION, where you supposedly need to answer
with a select-stream event.
bilboed: How soon are we supposed to be able to send the event? (I am porting
GES and I actually know it even before the pipeline start)
bilboed you don't *need* to answer with a SELECT_STREAM event you can just let
it do its automatic/default selection
slomo bilboed: yes, you might only care about video for example :)
thiblahute Well in case you want to select streeams... obviously :-) (And
forcing a special stream type is basically what I fixed in my last MR :-))
bilboed thiblahute, you can't really send that SELECT_STREAM event before
there's something to handle that event :) So yes, the earliest would be when a
collection is posted on the bus note that for GES you want something else in
addition you want to do *DISABLE* completely certain streams selecting streams
!= other streams are completely disabled
thiblahute Right, so yes it needs to be sync, which is not ideal for apps fmpov
(in GES it is not a big deal)
bilboed: Well at least no decoder is plugged, how do you completely disable
streams then?
slomo maybe there should be a flag that it does not default selection at all,
and waits for the application to do so?
bilboed a flag ... on what/where ?
slomo bilboed: not sure, on urisourcebin/dbin it would only stop those from
doing default selection... but wouldn't prevent e.g. a demuxer from doing a
default selection
bilboed also ... how would you prevent all the streaming threads from moving
forward ? By blocking them ? Like ... in a synchronous handler of some sort ? (a
bus message handler maybe ? :)
slomo bilboed: maybe some kind of GstContext on the pipeline, the standard
solution ;) that could also carry some default stream selection ("no audio
streams") or whatever if that seems useful to have
bilboed: if there was such a flag, i'd expect the elements to block until the
know which streams to output. not sure how well that works though, was just an
idea for how it could be easier for the application
bilboed why is it so hard to handle things in a synchronous message handler ?
thiblahute Well, the whole idea of the bus is to hide threads away from the app,
and fmpov we should try hard to make that a reality in all context
slomo multithreading is hard, applications often fail with getting these details
right in gstreamer. which is why we have the bus at all, why we have the
property notification messages (instead of just the notify signal), etc. the
videooverlay interface is an existing example where it makes things difficult
for applications
thiblahute (And I bet many people will answer from an async handler and not
understand what is going on)
slomo (and why we implemented it in playbin with caching and stuff so many
applications don't have to care)
generally we should try to not make it a requirement for new APIs that
applications have to handle things from random threads, more often than not they
get it wrong and things misbehave
thiblahute: i think a sync bus handler is also potentially problematic in python
with the GIL? it would kind of serialize the pipeline startup
* bilboed cries
thiblahute Yeah, I guess you could avoid that by connecting to
sync-message::stream-collection (or mitigate that at least)
bilboed ok, so how about this : have in playbin a sync handler delegator of
sorts
slomo thiblahute: indeed, that should work here. another footgun though :)
bilboed (note: "in playbin", not the default behaviour) requiring elements to do
more than just "post collection on bus, from that moment expect SELECT_STREAMS
event to potentially arrive" is going to be a burden slomo it would have to be
opt-in also (and even if just for backwards compatibility, the streams API is
stable) i don't know, needs some thinking. could also just be handled in
GstPlayer and if someone uses playbin directly we assume they know how to handle
threads
bilboed that too the problem is that the core (longer term) goal is to avoid any
un-needed processing/io/mem. And there's only one place/way you can do that : in
the element that can expose/handle those streams and *before* it decides what to
expose/output
slomo thiblahute: maybe create an issue for this so it's not forgotten? i don't
think we're going to come up with a solution now :)
bilboed: and another problem with it not outputting anything until the
STREAM_SELECT message is... that outputting something might be necessary
so that further downstream can know what streams it could output. or not?
bilboed "it" ?
__tim don't we have messages that you can post where the post() blocks til the
app has handled (or ignored) the message?
bilboed __tim, no we're not going down that road
slomo, the moment a stream provider (i.e. what posts collection and can handle
SELECT_STREAM events) has decided (or knows) what it's going to output, it posts
the STREAMS_SELECTED message on the bus
slomo __tim: yes in theory, but i don't think it works the way it would be
needed here. it's all a bit strange :)
bilboed slomo, let me triple-check, but it should be before any data is
outputted from db3
slomo bilboed: maybe a demuxer knows about a video stream, and posts the
message. further downstream at some point h264parse sees a CC stream in there
and extends the message. or container-in-container scenarios
bilboed slomo, that needs the capability for elements to notify the presence of
a stream within another (i.e. sub-streams) slomo, I have a plan for that
slomo bilboed: ack, i remember we talked about that at least once