part-buffering.txt 12.1 KB
Newer Older
1 2 3 4 5 6

This document outlines the buffering policy used in the GStreamer
core that can be used by plugins and applications.

The purpose of buffering is to accumulate enough data in a pipeline so that
Wim Taymans's avatar
Wim Taymans committed
playback can occur smoothly and without interruptions. It is typically done
9 10 11 12 13
when reading from a (slow) and non-live network source but can also be used for
live sources.

We want to be able to implement the following features:

Piotr Fusik's avatar
Piotr Fusik committed
 - buffering up to a specific amount of data, in memory, before starting playback
15 16 17 18 19 20 21 22
   so that network fluctuations are minimized.
 - download of the network file to a local disk with fast seeking in the
   downloaded data. This is similar to the quicktime/youtube players.
 - caching of semi-live streams to a local, on disk, ringbuffer with seeking in
   the cached area. This is similar to tivo-like timeshifting.
 - progress report about the buffering operations
 - easy (backward compatible) application notification of buffering
 - the possibility for the application to do more complex buffering

Some use cases:
Wim Taymans's avatar
Wim Taymans committed

26 27 28 29 30 31
 * Stream buffering:

   +---------+     +---------+     +-------+
   | httpsrc |     | buffer  |     | demux |
   |        src - sink      src - sink     ....
   +---------+     +---------+     +-------+
Wim Taymans's avatar
Wim Taymans committed
32 33

  In this case we are reading from a slow network source into a buffer element
  (such as queue2). 
Wim Taymans's avatar
Wim Taymans committed
35 36 37 38
  The buffer element has a low and high watermark expressed in bytes. The
  buffer uses the watermarks as follows:
39 40 41 42 43 44 45 46
   - The buffer element will post BUFFERING messages until the high watermark
     is hit. This instructs the application to keep the pipeline PAUSED, which
     will eventually block the srcpad from pushing while data is prerolled in
     the sinks.
   - When the high watermark is hit, a BUFFERING message with 100% will be
     posted, which instructs the application to continue playback.
   - When during playback, the low watermark is hit, the queue will start posting
     BUFFERING messages again, making the application PAUSE the pipeline again
Wim Taymans's avatar
Wim Taymans committed
47 48
     until the high watermark is hit again. This is called the rebuffering
49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81
   - during playback, the queue level will fluctuate between the high and the
     low watermark as a way to compensate for network irregularities.

  This buffering method is usable when the demuxer operates in push mode.
  Seeking in the stream requires the seek to happen in the network source.
  It is mostly desirable when the total duration of the file is not know, such
  as in live streaming or when efficient seeking is not possible/required.

 * Incremental download

   +---------+     +---------+     +-------+
   | httpsrc |     | buffer  |     | demux |
   |        src - sink      src - sink     ....
   +---------+     +----|----+     +-------+
  In this case, we know the server is streaming a fixed length file to the
  client. The application can choose to download the file on disk. The buffer
  element will provide a push or pull based srcpad to the demuxer to navigate in
  the downloaded file.

  This mode is only suitable when the client can determine the length of the
  file on the server.

  In this case, buffering messages will be emited as usual when the requested
  range is not within the downloaded area + buffersize. The buffering message
  will also contain an indication that incremental download is being performed.
  This flag can be used to let the application control the buffering in a more
  intelligent way, using the BUFFERING query, for example.

  The application can use the BUFFERING query to get the estimated download time
  and match this time to the current/remaining playback time to control when
Piotr Fusik's avatar
Piotr Fusik committed
  playback should start to have a non-interrupted playback experience.
83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112

 * Timeshifting

   +---------+     +---------+     +-------+
   | httpsrc |     | buffer  |     | demux |
   |        src - sink      src - sink     ....
   +---------+     +----|----+     +-------+

  In this mode, a fixed size ringbuffer is kept to download the server content.
  This allows for seeking in the buffered data. Depending on the size of the
  buffer one can seek further back in time.

  This mode is suitable for all live streams.

  As with the incremental download mode, buffering messages are emited along
  with an indication that timeshifting download is in progress. 

 * Live buffering

  In live pipelines we usually introduce some latency between the capture and
  the playback elements. This latency can be introduced by a queue (such as a
  jitterbuffer) or by other means (in the audiosink).

  Buffering messages can be emited in those live pipelines as well and serve as
  an indication to the user of the latency buffering. The application usually
  does not react to these buffering messages with a state change.
113 114 115


Wim Taymans's avatar
Wim Taymans committed
118 119 120 121 122 123
A GST_MESSAGE_BUFFERING must be posted on the bus when playback temporarily
stops to buffer and when buffering finishes. When percentage field in the
BUFFERING message is 100, buffering is done. Values less than 100 mean that
buffering is in progress. 

The BUFFERING message should be intercepted and acted upon by the application.
124 125 126 127 128 129 130 131 132 133
The message contains at least one field that is sufficient for basic

  "buffer-percent", G_TYPE_INT, between 0 and 100

Several more clever ways of dealing with the buffering messages can be used when
in incremental or timeshifting download mode. For this purpose additional fields
are added to the buffering message:

  "buffering-mode", GST_TYPE_BUFFERING_MODE, 
                   enum { "stream", "download", "timeshift", "live" }
      - gives the buffering mode in use. See above for an explanation of the
136 137
        different modes of buffering. This field can be used to let the
	application have more control over the buffering process.
138 139 140 141

  "avg-in-rate", G_TYPE_INT
      - gives the average input buffering speed in bytes/second. -1 is unknown.
        This is the average number of bytes per second that is received on the
	buffering element input (sink) pads. It is a measurement of the network
143 144 145 146 147 148 149
	speed in most cases.

  "avg-out-rate", G_TYPE_INT
      - gives the average consumption speed in bytes/second. -1 is unknown.
        This is the average number of bytes per second that is consumed by the
	downstream element of the buffering element. 

  "buffering-left", G_TYPE_INT64
151 152
      - gives the estimated time that bufferring will take in milliseconds.
        -1 unknown.
153 154 155
        This is measured based on the avg-in-rate and the filled level of the
        queue. The application can use this hint to update the GUI about the
        estimated remaining time that buffering will take.
Wim Taymans's avatar
Wim Taymans committed
156 157

Wim Taymans's avatar
Wim Taymans committed
159 160 161 162

While data is buffered, the pipeline should remain in the PAUSED state. It is
also possible that more data should be buffered while the pipeline is PLAYING,
in which case the pipeline should be PAUSED until the buffering finished.

Wim Taymans's avatar
Wim Taymans committed
164 165 166 167
BUFFERING messages can be posted while the pipeline is prerolling. The
application should not set the pipeline to PLAYING before a BUFFERING message
with 100 percent value is received, which might only happen after the pipeline

169 170 171 172 173 174 175 176 177 178
An exception is made for live pipelines. The application may not change
the state of a live pipeline when a buffering message is received. Usually these
buffering messages contain the "buffering-mode" = "live".

The buffering message can also instruct the application to switch to a periodical
BUFFERING query instead to more precisely control the buffering process. The
application can, for example, choose to not act on the BUFFERING message with
100 percent fill level to resume playback but instead use the estimated download
time to resume playback to get uninterrupted playback.


Buffering Query

183 184 185 186
In addition to the BUFFERING messages posted by the buffering elements we want
to be able to query the same information from the application. We also want to
be able to present the user with information about the downloaded range in the
file so that the GUI can react on it.

188 189
In addition to all the fields present in the buffering message, the BUFFERING
query contains the following field, which indicate the available downloaded
range in a specific format and the estimated time to complete:

192 193 194 195
  "busy", G_TYPE_BOOLEAN
    - if buffering was busy. This flag allows the application to pause the
      pipeline by using the query only.

196 197 198 199 200
  "format", GST_TYPE_FORMAT
    - the format of the "start" and "stop" values below  
  "start", G_TYPE_INT64, -1 unknown
    - the start position of the available data

202 203
  "stop", G_TYPE_INT64, -1 unknown
    - the stop position of the available data

205 206 207 208 209 210 211 212 213
  "estimated-total", G_TYPE_INT64
    - gives the estimated download time in milliseconds. -1 unknown.

    When the size of the downloaded file is known, this value will contain
    the latest estimate of the remaining download time. This value is usualy
    only filled for the "download" buffering mode. The application can use
    this information to estimate the amount of remaining time to download the
    complete file.

214 215 216
For the "download" and "timeshift" buffering-modes, the start and stop positions
specify the ranges where efficient seeking in the downloaded media is possible.
Seeking outside of these ranges might be slow or not at all possible.

218 219
For the "stream" and "live" mode the start and stop values describe the oldest
and newest item (expressed in "format") in the buffer. 


224 225 226 227 228 229

Some defaults for common elements:

A GstBaseSrc with random access replies to the BUFFERING query with:

  "buffer-percent" = 100
  "buffering-mode" = "stream"
231 232 233 234 235 236
  "avg-in-rate" = -1
  "avg-out-rate" = -1
  "buffering-left" = 0
  "format" = GST_FORMAT_BYTES
  "start" = 0
  "stop" = the total filesize
  "estimated-total" = 0
238 239 240 241

A GstBaseSrc in push mode replies to the BUFFERING query with:

  "buffer-percent" = 100
  "buffering-mode" = "stream"
243 244 245 246 247 248
  "avg-in-rate" = -1
  "avg-out-rate" = -1
  "buffering-left" = 0
  "format" = a valid GST_TYPE_FORMAT
  "start" = current position
  "stop" = current position
  "estimated-total" = -1

Wim Taymans's avatar
Wim Taymans committed
251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308

Buffering strategies

 Buffering strategies are specific implementations based on the buffering
 message and query described above.

 Most strategies have to balance buffering time versus maximal playback

 * simple buffering

   NON-live pipelines are kept in the paused state while buffering messages with
   a percent < 100% are received.

   This buffering strategy relies on the buffer size and low/high watermarks of
   the element. It can work with a fixed size buffer in memory or on disk.

   The size of the buffer is usually expressed in a fixed amount of time units
   and the estimated bitrate of the upstream source is used to convert this time
   to bytes.

   All GStreamer applications must implement this strategy. Failure to do so
   will result in starvation at the sink.

 * no-rebuffer strategy

   This strategy tries to buffer as much data as possible so that playback can
   continue without any further rebuffering.

   This strategy is initially similar to simple buffering, the difference is in
   deciding on the condition to continue playback. When a 100% buffering message
   has been received, the application will not yet start the playback but it will
   start a periodic buffering query, which will return the estimated amount of
   buffering time left. When the estimated time left is less than the remaining
   playback time, playback can continue.

   This strategy requires a unlimited buffer size in memory or on disk, such as
   provided by elements that implement the incremental download buffering mode.

   Usually, the application can choose to start playback even before the
   remaining buffer time elapsed in order to more quickly start the playback at
   the expense of a possible rebuffering phase.

 * Incremental rebuffering

   The application implements the simple buffering strategy but with each
   rebuffering phase, it increases the size of the buffer. 

   This strategy has quick, fixed time startup times but incrementally longer
   rebuffering times if the network is slower than the media bitrate.