part-framestep.txt 10.3 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Frame step
----------

This document outlines the details of the frame stepping functionality in
GStreamer.

The stepping functionality operates on the current playback segment, position
and rate as it was configured with a regular seek event. In contrast to the seek
event, it operates very closely to the sink and thus has a very low latency and
is not slowed down by queues and does not actually perform any seeking logic.
For this reason we want to include a new API instead of reusing the seek API.

The following requirements are needed:

 - The ability to walk forwards and backwards in the stream.
 - Arbitrary increments in any supported format (time, frames, bytes ...)
 - High speed, minimal overhead. This mechanism is not more expensive than
18
   simple playback. 
Stefan Kost's avatar
Stefan Kost committed
19
 - switching between forwards and backwards stepping should be fast.
20 21 22 23 24 25 26 27 28
 - Maintain synchronisation between streams.
 - Get feedback of the amount of skipped data.
 - Ability to play a certain amount of data at an arbitrary speed.

We want a system where we can step frames in PAUSED as well as play short
segments of data in PLAYING.


Use Cases
29
~~~~~~~~~
30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94

 * frame stepping in video only pipeline in PAUSED

    .-----.    .-------.              .------.    .-------.
    | src |    | demux |    .-----.   | vdec |    | vsink |
    |    src->sink    src1->|queue|->sink   src->sink     |
    '-----'    '-------'    '-----'   '------'    '-------'

    - app sets the pipeline to PAUSED to block on the preroll picture
    - app seeks to required position in the stream. This can be done with a
      positive or negative rate depending on the required frame stepping
      direction.
    - app steps frames (in GST_FORMAT_DEFAULT or GST_FORMAT_BUFFER). The
      pipeline loses its PAUSED state until the required number of frames have
      been skipped, it then prerolls again. This skipping is purely done in
      the sink.
    - sink posts STEP_DONE with amount of frames stepped and corresponding time
      interval.

 * frame stepping in audio/video pipeline in PAUSED

    .-----.    .-------.              .------.    .-------.
    | src |    | demux |    .-----.   | vdec |    | vsink |
    |    src->sink    src1->|queue|->sink   src->sink     |
    '-----'    |       |    '-----'   '------'    '-------'
               |       |              .------.    .-------.
               |       |    .-----.   | adec |    | asink |
               |      src2->|queue|->sink   src->sink     |
               '-------'    '-----'   '------'    '-------'


    - app sets the pipeline to PAUSED to block on the preroll picture
    - app seeks to required position in the stream. This can be done with a
      positive or negative rate depending on the required frame stepping
      direction.
    - app steps frames (in GST_FORMAT_DEFAULT or GST_FORMAT_BUFFER) or an amount
      of time on the video sink. The pipeline loses its PAUSED state until the
      required number of frames have been skipped, it then prerolls again.
      This skipping is purely done in the sink.
    - sink posts STEP_DONE with amount of frames stepped and corresponding time
      interval.
    - the app skips the same amount of time on the audiosink to align the
      streams again. When huge amount of video frames are skipped, there needs
      to be enough queueing in the pipeline to compensate for the accumulated
      audio.

 * frame stepping in audio/video pipeline in PLAYING

    - app sets the pipeline to PAUSED to block on the preroll picture
    - app seeks to required position in the stream. This can be done with a
      positive or negative rate depending on the required frame stepping
      direction.
    - app configures frames steps (in GST_FORMAT_DEFAULT or GST_FORMAT_BUFFER) or
      an amount of time on the sink. The step event has a flag indicating live
      stepping so that the stepping will only happens in PLAYING. 
    - app sets pipeline to PLAYING. The pipeline continues PLAYING until it
      consumed the amount of time.
    - sink posts STEP_DONE with amount of frames stepped and corresponding time
      interval. The sink will then wait for another step event. Since the
      STEP_DONE message was emited by the sink when it handed off the buffer to
      the device, there is usually sufficient time to queue a new STEP event so
      that one can seamlessly continue stepping.


events
95
~~~~~~
96

97 98
A new GST_EVENT_STEP event is introduced to start the step operation.  
The step event is created with the following fields in the structure:
99 100 101 102 103 104 105 106 107

  "format", GST_TYPE_FORMAT
     The format of the step units

  "amount", G_TYPE_UINT64
     The amount of units to step. -1 resumes normal non-stepping behaviour to
     the end of the segment.

  "rate", G_TYPE_DOUBLE
108 109 110 111 112 113 114
     The rate at which the frames should be stepped in PLAYING mode. 1.0 is
     the normal playback speed and direction of the segment, 2.0
     is double speed. A speed of 0.0 is not allowed. When performing a
     flushing step, the speed is not relevant. Note that we don't allow negative
     rates here, use a seek with a negative rate first to reverse the playback
     direction.

115
  "flush", G_TYPE_BOOLEAN
Piotr Fusik's avatar
Piotr Fusik committed
116
     when flushing is TRUE, the step is performed immediately:
117 118

      - In the PAUSED state the pipeline loses the PAUSED state, the requested
119 120 121
        amount of data is skipped and the pipeline prerolls again when a
	non-intermediate step completes.
	When the pipeline was stepping while the event is sent, the current step
122 123
	operation is updated with the new amount and format. The sink will do a
	best effort to comply with the new amount.
124 125 126 127 128
      - In the PLAYING state, the pipeline loses the PLAYING state, the
        requested amount of data is skipped (not rendered) from the previous STEP
	request or from the position of the last PAUSED if no previous STEP
	operation was performed. The pipeline goes back to the PLAYING state
	when a non-intermediate step completes.
129 130 131 132 133 134 135 136

     When flushing is FALSE, the step will be performed later.

      - In the PAUSED state the step will be done when going to PLAYING. Any
	previous step operation will be overridden with the new STEP event.
      - In the PLAYING state the step operation will be performed after the
	current step operation completes. If there was no previous step
	operation, the step operation will be performed from the position of the
Wim Taymans's avatar
Wim Taymans committed
137 138 139 140 141 142
	last PAUSED state. 

  "intermediate", G_TYPE_BOOLEAN
     Signal that this step operation is an intermediate step, part of a series
     of step operations. It is mostly interesting for stepping in the PAUSED state
     because the sink will only perform a preroll after a non-intermediate step
Piotr Fusik's avatar
Piotr Fusik committed
143
     operation completes. Intermediate steps are useful to flush out data from
Wim Taymans's avatar
Wim Taymans committed
144 145 146
     other sinks in order to not cause excessive queueing. In the PLAYING state
     the intermediate flag has no visual effect. In all states, the intermediate
     flag is passed to the corresponding GST_MESSAGE_STEP_DONE.
147 148


149 150 151
The application will create a STEP event to start or stop the stepping
operation. Both stepping in PAUSED and PLAYING can be performed by means of 
the flush flag.
152

153 154 155 156 157
The event is usually sent to the pipeline, which will typically distribute the
event to all of its sinks. For some use cases, like frame stepping on video
frames only, the event should only be sent to the video sink and upon reception
of the STEP_DONE message, one can step the other sinks to align the streams
again.
158

159 160 161
For large stepping amounts, there needs to be enough queueing in front of all
the sinks. If large steps need to be performed, they can be split up into
smaller step operations using the "intermediate" flag on the step. 
162

163 164 165
Since the step event does not update the base_time of any of the elements, the
sinks should keep track of the amount of stepped data in order to remain
synchronized against the clock.
166 167 168


messages
169
~~~~~~~~
170

171
A GST_MESSAGE_STEP_START is created. It contains the following fields.
172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190

  "active"
     If the step was queued or activated.

  "format", GST_TYPE_FORMAT
     The format of the step units that queued/activated.

  "amount", G_TYPE_UINT64
     The amount of units that were queued/activated.

  "rate", G_TYPE_DOUBLE
     The rate and direction at which the frames were queued/activated.

  "flush", G_TYPE_BOOLEAN
     If the queued/activated frames will be flushed.

  "intermediate", G_TYPE_BOOLEAN
     If this is an intermediate step operation that queued/activated.

191
The STEP_START message is emited 2 times:
192 193 194 195 196 197 198 199
 
  * first when an element received the STEP event and queued it. The "active"
    field will be FALSE in this case.

  * second when the step operation started in the streaming thread. The "active"
    field is TRUE in this case. After this message is emited, the application
    can queue a new step operation.

200 201 202
The purpose of this message is to find out how many elements participate in the
step operation and to queue new step operations at the earliest possible
moment.
203

204 205
A new GST_MESSAGE_STEP_DONE message is created. It contains the following
fields:
206 207 208 209 210 211 212 213 214 215

  "format", GST_TYPE_FORMAT
     The format of the step units that completed.

  "amount", G_TYPE_UINT64
     The amount of units that were stepped.

  "rate", G_TYPE_DOUBLE
     The rate and direction at which the frames were stepped.

216 217
  "flush", G_TYPE_BOOLEAN
     If the stepped frames were flushed.
218

Wim Taymans's avatar
Wim Taymans committed
219 220 221
  "intermediate", G_TYPE_BOOLEAN
     If this is an intermediate step operation that completed.

222 223 224
  "duration", G_TYPE_UINT64
     The total duration of the stepped units in GST_FORMAT_TIME.

225 226 227
  "eos", G_TYPE_BOOLEAN
     The step ended because of EOS.

228 229 230 231
The message is emited by the element that performs the step operation. The
purpose is to return the duration in GST_FORMAT_TIME of the stepped media. This
especially interesting to align other stream in case of stepping frames on the
video sink element.
232 233 234


Direction switch
235
~~~~~~~~~~~~~~~~
236

237 238
When quickly switching between a forwards and a backwards step of, for example,
one video frame, we need either:
239 240 241 242

  a) issue a new seek to change the direction from the current position.
  b) cache a certain number of stepped frames and walk the cache.

243 244 245 246 247
option a) might be very slow.
For option b) we would ideally like to offload this caching functionality to a
separate element, which means that we need to forward the STEP event upstream.
It's unclear how this could work in a generic way. What is a demuxer supposed
to do when it received a step event? a flushing seek to what stream position?
248