Feature requests for playbin/decodebin/uridecodebin and/or successors
Submitted by Carlos Rafael Giani
these are some notes about my experiences with these bins, and what features should be included in them, or in successor elements.
- Proper prebuffering/preparing in playbin
Right now, the gapless playback in playbin relies on downstream elements to have enough data buffered to cover the transition. In other words, playbin doesn't prepare the next stream before the current one ends. If the preparation isn't done very quickly, the queues downstream will eventually run out of data. In addition, if the stream is for example h.264 data, and a hardware h.264 decoder is to be used, an initial delay can occur, which would cause a noticeable stutter during the transition. This can be seen on embedded hardware like the i.MX6 devices.
I based my code on the concat element that is in GStreamer since version 1.6. With it, such prebuffering is possible. The downside is that a lot of steps that playbin does have to be done manually.
Prebuffering/preparing does introduce a new set of issues and necessities. First, preparing the next stream in the background causes an increase in CPU usage, since in that situation, two streams are active (though only one is producing output). This is particularly relevant on embedded devices with limited CPU power. It is therefore necessary to (a) prioritize the threads of the streams (give more CPU slices to the current stream) and (b) throttle the computations in the second stream, for example by limiting the input data rate in the streams with a probe and a condition variable inside that probe.
Second, if both the current and the next stream receive data over the network, then the current stream must be given priority. This can be solves in a similar way to (b) above, by throttling the data rate in the next stream.
- Make embedded devices first-class citizens during developments
Improvements in these bins and/or development of successors must involve embedded hardware during development, since a lot of situations where deadlocks, "clicks", stuttering etc. occur are not reproducible on more powerful machines such as PCs. I have seen many of such issues on a single core Sitara AM3352 machine with 720MHz, while the PC build showed none of these issues.
- Buffering improvements
"Buffering" in the bins is currently ill-defined and hardly introspectible. Stats such as the current buffer level are inaccessible. Buffer sizes/durations cannot be adjusted during playback. The buffering information must be a lot more detailed and be adjustable on the fly with more fine-grained controls. This is for example necessary with polluted Wi-Fi environments and HTTP transmissions of VBR data such as FLAC or Ogg Vorbis. Initial buffer size estimations can be way off with VBR, leading to frequent interruptions. A player's buffer logic could then resize the buffer based on observations made earlier. But right now, this is not doable with playbin/(uri)decodebin.
Comments are welcome. So are additional ideas for improvements.