playbin3: Add a mechanism to avoid requiring apps to connect synchronously to the bus to do stream selection
Right now the correct way to select streams with playbin3/decodebin3/uridecodebin3 is by connecting synchronously on the bus to listen to the
STREAM_COLLECTION signal, and from that callback, send a
SELECT_STREAMS event to the elements. While this work it is not ideal as the idea of having a pipeline bus is to hide threading away from the application, which is not the case here.
This was discussed on IRC and I am pasting here the conversation verbatim:
bilboed: Was the SELECT_STREAM message design to be answered on an async bus handler ? I observed that it was needed to be done synchronously which it a bit cumbersome for apps bilboed message ? There's only an event slomo you mean the STREAM_COLLECTION message? and based on that you'd like to do the initial SELECT_STREAM event to get a SELECTED_STREAMS message with the streams you wanted, without having the element first select its own default selection? then you need to handle it from a sync handler indeed bilboed and yes, ideally it should be answered synchronously. But if you *already* have preferences (like, you prefer french audio), it's quite fast to match, no ? you can always preroll first :) slomo that's rather awkward too though, you'd have to handle all the pads that appear. select things. pads might disappear, new ones might appear, etc. and you'd need to remove sinks potentially without deadlocking i think the only way to handle that cleanly is via a sync message handler for now bilboed pads might disappear ? Not unless you decide to disable a certain stream type (like no audio) thiblahute Right, I meant STREAM_COLLECTION, where you supposedly need to answer with a select-stream event. bilboed: How soon are we supposed to be able to send the event? (I am porting GES and I actually know it even before the pipeline start) bilboed you don't *need* to answer with a SELECT_STREAM event you can just let it do its automatic/default selection slomo bilboed: yes, you might only care about video for example :) thiblahute Well in case you want to select streeams... obviously :-) (And forcing a special stream type is basically what I fixed in my last MR :-)) bilboed thiblahute, you can't really send that SELECT_STREAM event before there's something to handle that event :) So yes, the earliest would be when a collection is posted on the bus note that for GES you want something else in addition you want to do *DISABLE* completely certain streams selecting streams != other streams are completely disabled thiblahute Right, so yes it needs to be sync, which is not ideal for apps fmpov (in GES it is not a big deal) bilboed: Well at least no decoder is plugged, how do you completely disable streams then? slomo maybe there should be a flag that it does not default selection at all, and waits for the application to do so? bilboed a flag ... on what/where ? slomo bilboed: not sure, on urisourcebin/dbin it would only stop those from doing default selection... but wouldn't prevent e.g. a demuxer from doing a default selection bilboed also ... how would you prevent all the streaming threads from moving forward ? By blocking them ? Like ... in a synchronous handler of some sort ? (a bus message handler maybe ? :) slomo bilboed: maybe some kind of GstContext on the pipeline, the standard solution ;) that could also carry some default stream selection ("no audio streams") or whatever if that seems useful to have bilboed: if there was such a flag, i'd expect the elements to block until the know which streams to output. not sure how well that works though, was just an idea for how it could be easier for the application bilboed why is it so hard to handle things in a synchronous message handler ? thiblahute Well, the whole idea of the bus is to hide threads away from the app, and fmpov we should try hard to make that a reality in all context slomo multithreading is hard, applications often fail with getting these details right in gstreamer. which is why we have the bus at all, why we have the property notification messages (instead of just the notify signal), etc. the videooverlay interface is an existing example where it makes things difficult for applications thiblahute (And I bet many people will answer from an async handler and not understand what is going on) slomo (and why we implemented it in playbin with caching and stuff so many applications don't have to care) generally we should try to not make it a requirement for new APIs that applications have to handle things from random threads, more often than not they get it wrong and things misbehave thiblahute: i think a sync bus handler is also potentially problematic in python with the GIL? it would kind of serialize the pipeline startup * bilboed cries thiblahute Yeah, I guess you could avoid that by connecting to sync-message::stream-collection (or mitigate that at least) bilboed ok, so how about this : have in playbin a sync handler delegator of sorts slomo thiblahute: indeed, that should work here. another footgun though :) bilboed (note: "in playbin", not the default behaviour) requiring elements to do more than just "post collection on bus, from that moment expect SELECT_STREAMS event to potentially arrive" is going to be a burden slomo it would have to be opt-in also (and even if just for backwards compatibility, the streams API is stable) i don't know, needs some thinking. could also just be handled in GstPlayer and if someone uses playbin directly we assume they know how to handle threads bilboed that too the problem is that the core (longer term) goal is to avoid any un-needed processing/io/mem. And there's only one place/way you can do that : in the element that can expose/handle those streams and *before* it decides what to expose/output slomo thiblahute: maybe create an issue for this so it's not forgotten? i don't think we're going to come up with a solution now :) bilboed: and another problem with it not outputting anything until the STREAM_SELECT message is... that outputting something might be necessary so that further downstream can know what streams it could output. or not? bilboed "it" ? __tim don't we have messages that you can post where the post() blocks til the app has handled (or ignored) the message? bilboed __tim, no we're not going down that road slomo, the moment a stream provider (i.e. what posts collection and can handle SELECT_STREAM events) has decided (or knows) what it's going to output, it posts the STREAMS_SELECTED message on the bus slomo __tim: yes in theory, but i don't think it works the way it would be needed here. it's all a bit strange :) bilboed slomo, let me triple-check, but it should be before any data is outputted from db3 slomo bilboed: maybe a demuxer knows about a video stream, and posts the message. further downstream at some point h264parse sees a CC stream in there and extends the message. or container-in-container scenarios bilboed slomo, that needs the capability for elements to notify the presence of a stream within another (i.e. sub-streams) slomo, I have a plan for that slomo bilboed: ack, i remember we talked about that at least once