part-qos.txt 15.8 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11
Quality-of-Service
------------------

Quality of service is about measuring and adjusting the real-time
performance of a pipeline.

The real-time performance is always measured relative to the pipeline
clock and typically happens in the sinks when they synchronize buffers
against the clock.

The measurements result in QOS events that aim to adjust the datarate
Piotr Fusik's avatar
Piotr Fusik committed
12
in one or more upstream elements. Two types of adjustments can be
13 14 15 16 17 18
made:

  - short time "emergency" corrections based on latest observation
    in the sinks.
  - long term rate corrections based on trends observed in the sinks.

Wim Taymans's avatar
Wim Taymans committed
19 20 21 22
It is also possible for the application to artificially introduce delay
between synchronized buffers, this is called throttling. It can be used
to reduce the framerate, for example.

23

24
Sources of quality problems
25
~~~~~~~~~~~~~~~~~~~~~~~~~~~
26 27

 - High CPU load
28
 - Network problems
29
 - Other resource problems such as disk load, memory bottlenecks etc.
Wim Taymans's avatar
Wim Taymans committed
30
 - application level throttling
31 32


33
QoS event
34
~~~~~~~~~
35

36 37
The QoS event is generated by an element that synchronizes against the clock. It
travels upstream and contains the following fields:
38

Wim Taymans's avatar
Wim Taymans committed
39 40 41 42 43 44 45 46 47 48 49 50 51
 - type, GST_TYPE_QOS_TYPE:
      The type of the QoS event, we have the following types and the default type
      is GST_QOS_TYPE_UNDERFLOW:

      GST_QOS_TYPE_OVERFLOW:  an element is receiving buffers too fast and can't
                              keep up processing them. Upstream should reduce the
                              rate.
      GST_QOS_TYPE_UNDERFLOW: an element is receiving buffers too slowly and has
                              to drop them because they are too late. Upstream should
                              increase the processing rate.
      GST_QOS_TYPE_THROTTLE:  the application is asking to add extra delay between
                              buffers, upstream is allowed to drop buffers

Wim Taymans's avatar
Wim Taymans committed
52 53 54 55
 - timestamp, G_TYPE_UINT64:
      The timestamp on the buffer that generated the QoS event. These timestamps
      are expressed in total running_time in the sink so that the value is ever
      increasing.
56

Wim Taymans's avatar
Wim Taymans committed
57 58 59
 - jitter, G_TYPE_INT64:
      The difference of that timestamp against the current clock time.  Negative
      values mean the timestamp was on time. Positive values indicate the
Wim Taymans's avatar
Wim Taymans committed
60 61
      timestamp was late by that amount. When buffers are received in time and 
      throttling is not enabled, the QoS type field is set to OVERFLOW.
Wim Taymans's avatar
Wim Taymans committed
62
      When throttling, the jitter contains the throttling delay added by the 
Wim Taymans's avatar
Wim Taymans committed
63
      application and the type is set to THROTTLE.
64

Wim Taymans's avatar
Wim Taymans committed
65 66 67
 - proportion, G_TYPE_DOUBLE:
      Long term prediction of the ideal rate relative to normal rate to get
      optimal quality.
68 69

The rest of this document deals with how these values can be calculated
70 71
in a sink and how the values can be used by other elements to adjust their
operations.
72 73


Wim Taymans's avatar
Wim Taymans committed
74
QoS message
75
~~~~~~~~~~~
Wim Taymans's avatar
Wim Taymans committed
76

Wim Taymans's avatar
Wim Taymans committed
77 78 79 80 81 82 83 84
A QOS message is posted on the bus whenever an element decides to:

 - drop a buffer because of QoS reasons
 - change its processing strategy because of QoS reasons (quality)

It should be expected that creating and posting the QoS message is reasonably
fast and does not significantly contribute to the QoS problems. Options to
disable this feature could also be presented on elements.
Wim Taymans's avatar
Wim Taymans committed
85 86

This message can be posted by a sink/src that performs synchronisation against the
Wim Taymans's avatar
Wim Taymans committed
87 88 89 90
clock (live) or it could be dropped by an upstream element that performs QoS
because of QOS events received from a downstream element (!live).

The GST_MESSAGE_QOS contains at least the following info:
Wim Taymans's avatar
Wim Taymans committed
91 92 93 94 95 96

 - live: G_TYPE_BOOLEAN:
      If the QoS message was dropped by a live element such as a sink or a live
      source. If the live property is FALSE, the QoS message was generated as a
      response to a QoS event in a non-live element.

Wim Taymans's avatar
Wim Taymans committed
97 98 99 100 101 102
 - running-time, G_TYPE_UINT64:
      The running_time of the buffer that generated the QoS message.

 - stream-time, G_TYPE_UINT64:
      The stream_time of the buffer that generated the QoS message.

Wim Taymans's avatar
Wim Taymans committed
103
 - timestamp, G_TYPE_UINT64:
Wim Taymans's avatar
Wim Taymans committed
104 105 106 107 108
      The timestamp of the buffer that generated the QoS message.

 - duration, G_TYPE_UINT64:
      The duration of the buffer that generated the QoS message.

Wim Taymans's avatar
Wim Taymans committed
109 110

 - jitter, G_TYPE_INT64:
Wim Taymans's avatar
Wim Taymans committed
111 112 113 114
      The difference of the running-time against the deadline.  Negative
      values mean the timestamp was on time. Positive values indicate the
      timestamp was late (and dropped) by that amount. The deadline can be
      a realtime running_time or an estimated running_time.
Wim Taymans's avatar
Wim Taymans committed
115 116

 - proportion, G_TYPE_DOUBLE:
Wim Taymans's avatar
Wim Taymans committed
117 118
      Long term prediction of the ideal rate relative to normal rate to get
      optimal quality.
Wim Taymans's avatar
Wim Taymans committed
119

Wim Taymans's avatar
Wim Taymans committed
120 121 122
 - quality, G_TYPE_INT:
      An element dependent integer value that specifies the current quality
      level of the element. The default maximum quality is 1000000. 
Wim Taymans's avatar
Wim Taymans committed
123 124


Wim Taymans's avatar
Wim Taymans committed
125 126 127 128
 - format, GST_TYPE_FORMAT
      Units of the 'processed' and 'dropped' fields. Video sinks and video
      filters will use GST_FORMAT_BUFFERS (frames). Audio sinks and audio filters
      will likely use GST_FORMAT_DEFAULT (samples).
Wim Taymans's avatar
Wim Taymans committed
129

Wim Taymans's avatar
Wim Taymans committed
130
 - processed: G_TYPE_UINT64:
Wim Taymans's avatar
Wim Taymans committed
131
      Total number of units correctly processed since the last state change to
Wim Taymans's avatar
Wim Taymans committed
132 133 134 135 136 137 138 139 140
      READY or a flushing operation.

 - dropped: G_TYPE_UINT64:
      Total number of units dropped since the last state change to READY or a
      flushing operation.

The 'running-time' and 'processed' fields can be used to estimate the average
processing rate (framerate for video).

Wim Taymans's avatar
Wim Taymans committed
141
Elements might add additional fields in the message which are documented in the
Wim Taymans's avatar
Wim Taymans committed
142
relevant elements or baseclasses.
Wim Taymans's avatar
Wim Taymans committed
143 144


145
Collecting statistics
146
~~~~~~~~~~~~~~~~~~~~~
147 148 149

A buffer with timestamp B1 arrives in the sink at time T1. The buffer
timestamp is then synchronized against the clock which yields a jitter J1
150
return value from the clock. The jitter J1 is simply calculated as
151

152
  J1 = CT - B1
153 154 155 156

Where CT is the clock time when the entry arrives in the sink. This value
is calculated inside the clock when we perform gst_clock_entry_wait().

157 158 159 160 161
If the jitter is negative, the entry arrived in time and can be rendered
after waiting for the clock to reach time B1 (which is also CT - J1).

If the jitter is positive however, the entry arrived too late in the sink 
and should therefore be dropped. J1 is the amount of time the entry was late.
162

163
Any buffer that arrives in the sink should generate a QoS event upstream.
164 165 166 167

Using the jitter we can calculate the time when the buffer arrived in the
sink: 
     
168
  T1 = B1 + J1.                                (1)
169 170 171

The time the buffer leaves the sink after synchronisation is measured as:

172
  T2 = B1 + (J1 < 0 ? 0 : J1)                  (2)
173

174 175 176
For buffers that arrive in time (J1 < 0) the buffer leaves after synchronisation
which is exactly B1. Late buffers (J1 >= 0) leave the sink when they arrive,
whithout any synchronisation, which is T2 = T1 = B1 + J1.
177 178 179 180

Using a previous T0 and a new T1, we can calculate the time it took for
upstream to generate a buffer with timestamp B1.

181
  PT1 = T1 - T0                                (3)
182 183 184 185 186 187 188 189 190 191 192 193 194

We call PT1 the processing time needed to generate buffer with timestamp B1.

Moreover, given the duration of the buffer D1, the current data rate (DR1) of 
the upstream element is given as:

       PT1   T1 - T0
 DR1 = --- = -------                           (4)
       D1      D1

For values 0.0 < DR1 <= 1.0 the upstream element is producing faster than
real-time. If DR1 is exactly 1.0, the element is running at a perfect speed.

195
Values DR1 > 1.0 mean that the upstream element cannot produce buffers of
196 197 198
duration D1 in real-time. It is exactly DR1 that tells the amount of speedup
we require from upstream to regain real-time performance.

Wim Taymans's avatar
Wim Taymans committed
199
An element that is not receiving enough data is said to be underflowed.
200 201 202


Element measurements
203
~~~~~~~~~~~~~~~~~~~~
204 205 206 207 208

In addition to the measurements of the datarate of the upstream element, a
typical element must also measure its own performance. Global pipeline 
performance problems can indeed also be caused by the element itself when it
receives too much data it cannot process in time. The element is then said to
Wim Taymans's avatar
Wim Taymans committed
209
be overflowed.
210

211 212 213 214 215 216 217 218 219

Short term correction
---------------------

The timestamp and jitter serve as short term correction information
for upstream elements. Indeed, given arrival time T1 as given in (1)
we can be certain that buffers with a timestamp B2 < T1 will be too late
in the sink.

220
In case of a positive jitter we can therefore send a QoS event with
221 222
a timestamp B1, jitter J1 and proportion given by (4).

223
This allows an upstream element to not generate any data with timestamps
224
B2 < T1, where the element can derive T1 as B1 + J1.
225 226 227 228 229 230

This will effectively result in frame drops.

The element can even do a better estimation of the next valid timestamp it
should output.

231
Indeed, given the element generated a buffer with timestamp B0 that arrived
232
in time in the sink but then received a QoS event stating B1 arrived J1
233
too late. This means generating B1 took (B1 + J1) - B0 = T1 - T0 = PT1, as 
234 235 236
given in (3). Given the buffer B1 had a duration D1 and assuming that
generating a new buffer B2 will take the same amount of processing time,
a better estimation for B2 would then be:
237

238
  B2 = T1 + D2 * DR1
239 240 241

 expanding gives:

242
  B2 = (B1 + J1) + D2 * (B1 + J1 - B0)
243
                        --------------  
244
		             D1
245

246
 assuming the durations of the frames are equal and thus D1 = D2:
247

248
  B2 = (B1 + J1) + (B1 + J1 - B0)
249

250
  B2 =  2 * (B1 + J1) - B0
251 252 253

 also:

254
  B0 = B1 - D1
255 256 257

 so:

258
  B2 =  2 * (B1 + J1) - (B1 - D1)
259 260 261

Which yields a more accurate prediction for the next buffer given as:

262
  B2 =  B1 + 2 * J1 + D1                          (5)
263 264 265 266 267 268 269 270 271 272 273 274 275


Long term correction
--------------------

The datarate used to calculate (5) for the short term prediction is based
on a single observation. A more accurate datarate can be obtained by
creating a running average over multiple datarate observations.

This average is less susceptible to sudden changes that would only influence
the datarate for a very short period.

A running average is calculated over the observations given in (4) and is
276
used as the proportion member in the QoS event that is sent upstream.
277

278
Receivers of the QoS event should permanently reduce their datarate 
279
as given by the proportion member. Failure to do so will certainly lead to
280
more dropped frames and a generally worse QoS.
281 282


Wim Taymans's avatar
Wim Taymans committed
283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300
Throttling
----------

In throttle mode, the time distance between buffers is kept to a configurable
throttle interval. This means that effectively the buffer rate is limited
to 1 buffer per throttle interval. This can be used to limit the framerate,
for example.

When an element is configured in throttling mode (this is usually only
implemented on sinks) it should produce QoS events upstream with the jitter
field set to the throttle interval. This should instruct upstream elements to
skip or drop the remaining buffers in the configured throttle interval.

The proportion field is set to the desired slowdown needed to get the
desired throttle interval. Implementations can use the QoS Throttle type,
the proportion and the jitter member to tune their implementations. 


301 302 303
QoS strategies
--------------

304 305 306
Several strategies exist to reduce processing delay that might affect
real time performance.

307
 - lowering quality
308 309 310 311 312 313 314 315
   - dropping frames (reduce CPU/bandwidth usage)
   - switch to a lower decoding/encoding quality (reduce algorithmic
     complexity)
   - switch to a lower quality source (reduce network usage)
 - increasing thread priorities
   - switch to real-time scheduling
   - assign more CPU cycles to critial pipeline parts
   - assign more CPU(s) to critical pipeline parts
316

317

318 319 320
QoS implementations
-------------------

321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338
Here follows a small overview of how QoS can be implemented in a range of
different types of elements.


GstBaseSink
-----------

The primary implementor of QoS is GstBaseSink. It will calculate the following
values:

 - upstream running average of processing time (5) in stream time.
 - running average of buffer durations.
 - running average of render time (in system time)
 - rendered/dropped buffers

The processing time and the average buffer durations will be used to
calculate a proportion.

339
The processing time in system time is compared to render time to decide if
340
the majority of the time is spend upstream or in the sink itself. This value
Wim Taymans's avatar
Wim Taymans committed
341
is used to decide overflow or underflow.
342

343
The number of rendered and dropped buffers is used to query stats on the sink.
344

345
A QoS event with the most current values is sent upstream for each buffer 
346 347 348
that was received by the sink.

Normally QoS is only enabled for video pipelines. The reason being that drops
349
in audio are more disturbing than dropping video frames. Also video requires in 
350 351 352
general more processing than audio.

Normally there is a threshold for when buffers get dropped in a video sink. Frames
Piotr Fusik's avatar
Piotr Fusik committed
353
that arrive 20 milliseconds late are still rendered as it is not noticeable for
354 355
the human eye.

356 357
A QoS message is posted whenever a (part of a) buffer is dropped.

Wim Taymans's avatar
Wim Taymans committed
358 359 360 361 362
In throttle mode, the sink sends QoS event upstream with the timestamp set to
the running_time of the latest buffer and the jitter set to the throttle interval.
If the throttled buffer is late, the lateness is subtracted from the throttle
interval in order to keep the desired throttle interval. 

363 364 365 366 367

GstBaseTransform
----------------

Transform elements can entirely skip the transform based on the timestamp and
368
jitter values of recent QoS event since these buffers will certainly arrive
369 370 371 372 373 374 375
too late.

With any intermediate element, the element should measure its performance to
decide if it is responsible for the quality problems or any upstream/downstream
element.

some transforms can reduce the complexity of their algorithms. Depending on the
376
algorithm, the changes in quality may have disturbing visual or audible effect 
377 378
that should be avoided.

379
A QoS message should be posted when a frame is dropped or when the quality
380 381 382
of the filter is reduced. The quality member in the QOS message should reflect
the quality setting of the filter.

383 384 385 386 387 388 389 390 391

Video Decoders
--------------

A video decoder can, based on the codec in use, decide to not decode intermediate
frames. A typical codec can for example skip the decoding of B-frames to reduce
the CPU usage and framerate.

If each frame is independantly decodable, any arbitrary frame can be skipped based
392
on the timestamp and jitter values of the latest QoS event. In addition can the
393
proportion member be used to permanently skip frames.
394

395 396
It is suggested to adjust the quality field of the QoS message with the expected
amount of dropped frames (skipping B and/or P frames). This depends on the
Wim Taymans's avatar
Wim Taymans committed
397
particular spacing of B and P frames in the stream. If the quality control would
398
result in half of the frames to be dropped (typical B frame skipping), the
Wim Taymans's avatar
Wim Taymans committed
399 400 401 402
quality field would be set to 1000000 * 1/2 = 500000. If a typical I frame spacing
of 18 frames is used, skipping B and P frames would result in 17 dropped frames
or 1 decoded frame every 18 frames. The quality member should be set to
1000000 * 1/18 = 55555.
403 404

 - skipping B frames: quality = 500000
Wim Taymans's avatar
Wim Taymans committed
405
 - skipping P/B frames: quality = 55555 (for I-frame spacing of 18 frames)
406

407 408 409 410

Demuxers
--------

411
Demuxers usually cannot do a lot regarding QoS except for skipping frames to the next
412
keyframe when a lateness QoS event arrives on a source pad.
413 414

A demuxer can however measure if the performance problems are upstream or downstream
415
and forward an updated QoS event upstream.
416

417 418
Most demuxers that have multiple output pads might need to combine the QoS
events on all the pads and derive an aggregated QoS event for the upstream element.
419 420 421 422 423


Sources
-------

424
The QoS events only apply to push based sources since pull based sources are entirely
425 426
controlled by another downstream element.

Wim Taymans's avatar
Wim Taymans committed
427
Sources can receive a overflow or underflow event that can be used to switch to
428 429 430 431
less demanding source material. In case of a network stream, a switch could be done
to a lower or higher quality stream or additional enhancement layers could be used
or ignored.

432
Live sources will automatically drop data when it takes too long to process the data 
433 434 435
that the element pushes out. 

Live sources should post a QoS message when data is dropped.
436