part-negotiation.txt 11.3 KB
Newer Older
Wim Taymans's avatar
Wim Taymans committed
1 2 3

4 5 6 7 8 9 10 11
Capabilities negotiation is the process of deciding on an adequate
format for dataflow within a GStreamer pipeline. Ideally, negotiation
(also known as "capsnego") transfers information from those parts of the
pipeline that have information to those parts of the pipeline that are
flexible, constrained by those parts of the pipeline that are not

GStreamer's two scheduling modes, push mode and pull mode, lend
themselves to different mechanisms to achieve this goal. As it is more
13 14 15
common we describe push mode negotiation first.

Push-mode negotiation

Push-mode negotiation happens when elements want to push buffers and
19 20 21
need to decide on the format. This is called downstream negotiation
because the upstream element decides the format for the downstream
element. This is the most common case.
Wim Taymans's avatar
Wim Taymans committed
22 23 24 25 26 27 28

Negotiation can also happen when a downstream element wants to receive 
another data format from an upstream element. This is called upstream

The basics of negotiation are as follows:

Wim Taymans's avatar
Wim Taymans committed
 - GstCaps (see part-caps.txt) are refcounted before they
Wim Taymans's avatar
Wim Taymans committed
30 31 32 33 34 35 36 37 38
   are attached to a buffer to describe the contents of the buffer.
   It is possible to add a NULL caps to a buffer, this means that the
   buffer type did not change relative to the previous buffer. If no
   previous buffer was received by a downstream element, it is free to
   discard the buffer.

 - Before receiving a buffer, an element must check if the datatype of
   the buffer has changed. The element should reconfigure itself to the
   new format before processing the buffer data. If the data type on
Wim Taymans's avatar
Wim Taymans committed
   the buffer is not acceptable, the element should refuse the buffer by 
40 41
   returning an appropriate GST_FLOW_NOT_NEGOTIATED return value from the
   chain function.
42 43
   The core will automatically call the set_caps function for this purpose
   when it is installed on the sink or source pad.
Wim Taymans's avatar
Wim Taymans committed

Piotr Fusik's avatar
Piotr Fusik committed
 - When requesting a buffer from a bufferpool, the preferred type should
Wim Taymans's avatar
Wim Taymans committed
46 47 48 49
   be passed to the buffer allocation function. After receiving a buffer
   from a bufferpool, the datatype should be checked again.

 - A bufferpool allocation function should try to allocate a buffer of the
Piotr Fusik's avatar
Piotr Fusik committed
   preferred type. If there is a good reason to choose another type, the
Wim Taymans's avatar
Wim Taymans committed
51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101
   alloc function should see if that other type is accepted by the other
   element, then allocate a buffer of that type and attach the type to the
   buffer before returning it.

The general flow for a source pad starting the negotiation.

             src              sink
              |                 |
              |  accepts?       |
  type A      |---------------->|
              |      yes        |
              |                 |
 get buffer   |  alloc_buf      |
 from pool    |---------------->| 
 with type A  |                 | Create buffer of type A.
              |                 |
 check type   |<----------------|
 and use A    |                 |
              |  push           |
 push buffer  |---------------->| Receive type A, reconfigure to
 with new type|                 | process type A.
              |                 |

 One possible implementation in pseudo code:

 [element wants to create a buffer]
 if not format
   # see what the peer can do
   peercaps = gst_pad_peer_get_caps (srcpad)
   # see what we can do
   ourcaps = gst_pad_get_caps (srcpad)

   # get common formats
   candidates = gst_caps_intersect (peercaps, ourcaps)

   foreach candidate in candidates
     # make sure the caps is fixed
     fixedcaps = gst_pad_fixate_caps (srcpad, candidate)

     # see if the peer accepts it
     if gst_pad_peer_accept_caps (srcpad, fixedcaps)
       # store the caps as the negotiated caps, this will
       # call the setcaps function on the pad
       gst_pad_set_caps (srcpad, fixedcaps)

102 103
 # if the type is different, the buffer will have different caps from
 # the src pad -- setcaps will get called on the pad_push
Wim Taymans's avatar
Wim Taymans committed
104 105 106 107 108 109 110 111 112 113 114 115 116
 buffer = gst_pad_alloc_buffer (srcpad, 0, size, GST_PAD_CAPS (fixedcaps));
 if buffer
   [fill buffer and push]
   [no buffer, either no peer or no acceptable format found]

The general flow for a sink pad starting a renegotiation.

             src              sink
              |                 |
              |  accepts?       |
Stefan Kost's avatar
Stefan Kost committed
              |<----------------| type B
Wim Taymans's avatar
Wim Taymans committed
118 119 120 121 122
              |      yes        |
              |                 |
 get buffer   |  alloc_buf      |
 from pool    |---------------->| 
Wim Taymans's avatar
Wim Taymans committed
 with type A  |                 | Create buffer of new type B.
Wim Taymans's avatar
Wim Taymans committed
124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169
              |                 |
 check type   |<----------------|
 and          |                 |
 reconfigure  |                 |
              |  push           |
 push buffer  |---------------->| Receive type B, reconfigure to
 with new type|                 | process type B.
              |                 |


Use case:

videotestsrc ! xvimagesink

  1) Who decides what format to use?
   - src pad always decides, by convention. sinkpad can suggest a format
     by putting it high in the getcaps function GstCaps. 
   - since the src decides, it can always choose something that it can do,
     so this step can only fail if the sinkpad stated it could accept
     something while later on it couldn't.

  2) When does negotiation happen?
   - before srcpad does a push, it figures out a type as stated in 1), then 
     it calls the pad alloc function with the type. The sinkpad has to 
     create a buffer of that type, src fills the buffer and sends it to sink.
   - since the sink stated in 1) it could accept the type, it will be able to
     create a buffer of the type and handle it.
   - sink checks media type of buffer and configures itself for this type.
  3) How can sink request another format?
   - sink asks if new format is possible for the source.
   - sink returns buffer with new type in allocfunction.
   - src receives buffer with new type, reconfigures and pushes.
   - sink can always select something it can create and handle since it takes
     the initiative. src should be able to handle the new type since it said
     it could accept it.

videotestsrc ! queue ! xvimagesink

  - queue implements an allocfunction, proxying all calls to its srcpad peer.
  - queue proxies all accept and getcaps to the other peer pad.
  - queue contains buffers with different types.

Pull-mode negotiation


176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205
A pipeline in pull mode has different negotiation needs than one
activated in push mode. Push mode is optimized for two use cases:

 * Playback of media files, in which the demuxers and the decoders are
   the points from which format information should disseminate to the
   rest of the pipeline; and

 * Recording from live sources, in which users are accustomed to putting
   a capsfilter directly after the source element; thus the caps
   information flow proceeds from the user, through the potential caps
   of the source, to the sinks of the pipeline.

In contrast, pull mode has other typical use cases:

 * Playback from a lossy source, such as RTP, in which more knowledge
   about the latency of the pipeline can increase quality; or

 * Audio synthesis, in which audio APIs are tuned to producing only the
   necessary number of samples, typically driven by a hardware interrupt
   to fill a DMA buffer or a Jack[0] port buffer.

 * Low-latency effects processing, whereby filters should be applied as
   data is transferred from a ring buffer to a sink instead of
   beforehand. For example, instead of using the internal alsasink
   ringbuffer thread in push-mode wavsrc ! volume ! alsasink, placing
   the volume inside the sound card writer thread via wavsrc !
   audioringbuffer ! volume ! alsasink.


The problem with pull mode is that the sink has to know the format in
207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222
order to know how many bytes to pull via gst_pad_pull_range(). This
means that before pulling, the sink must initiate negotation to decide
on a format.

Recalling the principles of capsnego, whereby information must flow from
those that have it to those that do not, we see that the two named use
cases have different negotiation requirements:

 * RTP and low-latency playback are both like the normal playback case,
   in which information flows downstream.

 * In audio synthesis, the part of the pipeline that has the most
   information is the sink, constrained by the capabilities of the graph
   that feeds it. However the caps are not completely specified; at some
   point the user has to intervene to choose the sample rate, at least.
   This can be done externally to gstreamer, as in the jack elements, or
   internally via a capsfilter, as is customary with live sources.
224 225 226

Given that sinks potentially need the input of sources, as in the RTP
case and at least as a filter in the synthesis case, there must be a
227 228 229 230 231
negotiation phase before the pull thread is activated. Also, given the
low latency offered by pull mode, we want to avoid capsnego from within
the pulling thread, in case it causes us to miss our scheduling

232 233 234 235 236
The pull thread is usually started in the PAUSED->PLAYING state change. We must
be able to complete the negotiation before this state change happens.

The time to do capsnego, then, is after _check_pull_range() has succeeded,
but before the sink has spawned the pulling thread.
237 238 239


242 243 244
The sink determines that the upstream elements support pull based scheduling by
calling gst_pad_check_pull_range().

245 246 247 248 249 250 251 252 253
The sink initiates the negotiation process by intersecting the results
of gst_pad_get_caps() on its sink pad and its peer src pad. This is the
operation performed by gst_pad_get_allowed_caps(). In the simple
passthrough case, the peer pad's getcaps() function should return the
intersection of calling get_allowed_caps() on all of its sink pads. In
this way the sink element knows the capabilities of the entire pipeline.

The sink element then fixates the resulting caps, if necessary,
resulting in the flow caps. It notifies the pipeline of the caps by
254 255 256
calling gst_pad_set_caps() on its sink pad. From now on, the getcaps function
of the sinkpad will only return these fixed caps meaning that upstream elements
will only be able to produce this format.
257 258 259 260 261

If the sink element could not set caps on its sink pad, it should post
an error message on the bus indicating that negotiation was not

262 263 264 265 266 267 268 269 270 271
When negotiation succeeded, the sinkpad and all upstream internally linked pads
are activated in pull mode. Typically, this operation will trigger negotiation
on the downstream elements, which will now be forced to negotiation to the
final fixed desired caps of the sinkpad.

After these steps, the sink element returns ASYNC from the state change
function. The state will commit to PAUSED when the first buffer is received in
the sink. This is needed to provide a consistent API to the applications that
expect ASYNC return values from sinks but it also allows us to perform the
remainder of the negotiation outside of the context of the pulling thread.
272 273 274 275 276 277 278

During dataflow, gst_pad_pull_range() checks the caps on the pulled
buffer. If they are different from the sink pad's caps, it will return
GST_FLOW_NOT_NEGOTIATED. Because of the low-latency requirements,
changing caps in an activate pull-mode pipeline is not supported, as it
might require e.g. the sound card to reconfigure its hardware buffers,
and start capsnego again.