Question: tee, appsink and offset
I tried implementing the tee
as you suggested and I'm nearly done. I have two problems left that are probably related.
For easier reference, this is the pipeline (for an audio only media):
--- queue (visualization) --- audioconvert --- appsink (-> callback to visualization)
src --- tee /
\
--- queue (playback) --- audioconvert --- audioresample --- autoaudiosink
In my visualization, I want to display 1 sec of samples ahead of current position and 1 sec behind (this might change depending on the zoom factor). See this screenshot as an illustration. I read the tutorials on multithreading, cutting the pipeline and streaming and ended up setting:
- source element (
uridecodebin
):"buffer-duration"
property to 3 sec (as nano). - visualization
queue
:"max-size-time"
property to 3 sec (as nano). I believe the defaults for the other properties should be good as is ("max-size-bytes"
being 10MB and"max-size-buffers"
200). - visualization
queue
:offset
to -1 sec (as nano). -
appsink
:offset
to -1 sec (as nano).
I checked the appsink example and implemented appsink.set_callbacks
to get the samples. I first start my pipeline
in the Paused
state, so I defined both the new_preroll
and new_sample
closures.
Problem # 1
After the pipeline
is set to Paused
, I only get the first sample
in the new_preroll
closure. I also defined a pad probe on the queue
sink
which gives me the expected many buffers. So I guess the appsink
doesn't pull the buffers
during preroll
even though they are available. I checked the properties
for appsink
, but the defaults seem to fit my needs.
What am I missing or doing wrong? I guess I would be better off implementing the visualization as a plugin, but it is supposed to be interactive and will be really tied up to the application.
Problem # 2
When I define the offset
for the visualization queue
and appsink
to a value less than -1 sec (so further than 1 sec from 0), I no longer can get the state
of the pipeline
using let (_, current, _) = self.pipeline.get_state(10_000_000);
. I tried larger values than 10 ms, but it didn't help.
Observation
Finally, not a show stopper but rather an observation. In the callbacks for pad probe or those from the appsink
, the buffers' offset
and offset_end
are always equal and constant. According to this page, I was expecting to be able to rely on them to get the position of the buffer in the stream. I searched in gstreamer-rs
's code, but found nothing suspicious there.