Commit 866fefbf authored by Tim-Philipp Müller's avatar Tim-Philipp Müller 🐠
Browse files

docs: design: remove, moved to gst-docs

parent b87e5b9c
design-rtpauxiliary.txt \
design-rtcollision.txt \
RTP auxiliary stream design
auxiliary elements
There are two kind of auxiliary elements, sender and receiver.
Let's call them rtpauxsend and rtpauxreceive.
rtpauxsend has always one sink pad and can have unlimited requested src pads.
If only src pad then it works in SSRC-multiplexed mode, if several src pads
then it works in session multiplexed mode.
rtpauxreceive has always one ssrc pad and can have unlimited requested sink pads.
If only one sink pad then it works in SSRC-multiplexed mode, if several sink pads
then it works in session multiplexed mode.
rtpbin and auxiliary elements
-- basic mecanism
rtpbin knows for which session ids the given auxiliary element belong to.
It's done through "set-aux-send", for rtpauxsend kind, and through
"set-aux-receive" for rtpauxreceive kind.
You can call those signals as much as needed for each auxiliary element.
So for aux elements that work in SSRC-multiplexed mode this signal action is
called only one time.
The user has to call those action signals before to request the differents
rtpbin pads.
rtpbin is in charge to link those auxiliary elements with the sessions,
and on receiver side, rtpbin has also to handle the link with ssrcdemux.
rtpbin never knows if the given rtpauxsend is actually a rtprtxsend element
or another aux element.
rtpbin never knows if the given rtpauxreceive is actually a rtprtxreceive
element or another aux element.
rtpbin has to be kept generic so that more aux elements can be added later
without changing rtpbin.
It's currently not possible to use rtpbin with auxiliary stream from gst-launch.
We can discuss about having the ability for rtpbin to instanciate itself
the special aux elements rtprtxsend and rtprtxreceive but they need to be
configured ("payload-type" and "payload-types" properties) to make retransmission
work. So having several rtprtxsend and rtprtxreceive in a rtpbin would require
a lot of properties to manage them form rtpbin.
And for each auxiliary elements.
If you want to use rtprtxreceive and rtprtpsend from gst-launch you have to use
rtpsession, ssrcdemux and rtpjitterbuffer elements yourself.
See gtk-doc of rtprtxreceive for an example.
-- requesting the rtpbin's pads on the pipeline receiver side
If rtpauxreceive is set for session, i, j, k then it has to call
rtpbin::"set-aux-receive" 3 times giving those ids and this aux element.
It has to be done before requesting the recv_rtp_sink_i, recv_rtp_sink_j, recv_rtp_sink_k.
For a concrete case rtprtxreceive, if the user wants it for session i, then it has
to call rtpbin::"set-aux-receive" one time giving i and this aux element.
Then the user can request recv_rtp_sink_i pad.
Calling rtpbin::"set-aux-receive" does not create the session. It add the given
session id and aux element to a hashtable(key:session id, value: aux element).
Then when the user ask for rtpbin.recv_rtp_sink_i, rtpbin lookup if there is an
aux element for this i session id. If yes it requests a sink pad to this aux
element and links it with the recv_rtp_src pad of the new gstrtpsession.
rtpbin also checks that this aux element is connected only one time to ssrcdemux.
Because rtpauxreceive has only one source pad.
Each call to request rtpbin.recv_rtp_sink_k will also creates rtpbin.recv_rtp_src_k_ssrc_pt
as usual. So that the user have it when then it requests rtpbin. (from gst-launch) or
using on_rtpbinreceive_pad_added callback from an application.
-- requesting the rtpbin's pads on the pipeline sender side
For the sender this is similar but a bit more complicated to implement.
When the user asks for rtpbin.send_rtp_sink_i, rtpbin will lookup in its
second map (key:session id, value: aux send element). If there is one aux
element, then it will set the sink pad of this aux sender element to be the ghost
pad rtpbin.send_rtp_sink_i that the user asked. rtpbin will also request a src
pad of this aux element to connect it to gstrtpsession_i. It will automatically
create rtpbin.send_rtp_src_i the usuall way. Then if the user asks
rtpbin.send_rtp_src_k, then rtpbin will also lookup in that map and request
another source pad of the aux element and connect it to the new gstrtpsession_k.
RTP collision design
Custon upstream event which contains the ssrc marked as collided.
This event is generated on both pipeline sender and receiver side by
the gstrtpsession element when it detects a conflict between ssrc.
(same session id and same ssrc)
It's an upstream event so that means this event is for now only
useful on pipeline sender side. Because elements generating packets with the
collided SSRC are placed upstream from the gstrtpsession.
When handling a GstRTPCollision event, the rtppayloader has to choose another
BYE only the corresponding source, not the whole session.
When a collision happens for the given ssrc, the associated source is marked
bye. But we make sure that the whole session is not itself set bye.
Because internally, gstrtpsession can manages several sources and all have
their own distinct ssrc.
RTP retransmission design
Custom upstream event which mainly contains the ssrc and the seqnum of the
packet which is asked to be retransmisted.
On the pipeline receiver side this event is generated by the
gstrtpjitterbuffer element. Then it is translated to a NACK to be sent over
the network.
On the pipeline sender side, this event is generated by the gstrtpsession
element when it receives a NACK from the network.
rtprtxsend element
-- basic mechanism
rtprtxsend keeps a history of rtp packets that it has already sent.
When it receives the event GstRTPRetransmissionRequest from the downstream
gstrtpsession element, it loopkup the requested seqnum in its stored packets.
If the packet is present in its history, it will create a RTX packet according
to RFC 4588. Then this rtx packet is pushed to its src pad as other packets.
rtprtxsend works in SSRC-multiplexed mode, so it has one always sink and
src pad.
-- building retransmission packet fron original packet
A rtx packet is mostly the same as an orignal packet, except it has its own
ssrc and its own seqnum. That's why rtprtxsend works in SSRC-multiplexed mode.
It also means that the same session is used.
Another difference between rtx packet and its original is that it inserts the
original seqnum (OSN: 2 bytes) at the beginning of the payload.
Also rtprtxsend builds rtx packet without padding, to let other elements do that.
The last difference is the payload type. For now the user has to set it through
the rtx-payload-type property. Later it will be automatically retreive this
information from SDP. See fmtp field as specifies in the RPC4588
(a=fmtp:99 apt=98) fmtp is the payload type of the retransmission stream
and apt the payload type of its associated master stream.
-- restransmission ssrc and seqnum
To choose rtx_ssrc it randomly selects a number between 0 and 2^32-1 until
it is different than master_ssrc.
rtx_seqnum is randomly selected between 0 and 2^16-1
-- deeper in the stored buffer history
For the history it uses a GSequence with 2^15-1 as its maximum size.
Which is resonable as the default value is 100.
It contains the packets in reverse order they have been sent
(head:newest, tail:oldest)
GSequence allows to add and remove an element in constant time (like a queue).
Also GSequence allows to do a binary search when rtprtxsend lookup in its
It's important if it receives a lot of requests or if the history is large.
-- pending rtx packets
When looking up in its history, if seqnum is found then it pushes the buffer
into a GQueue to its tail.
Before to send the current master stream packet, rtprtxsend sends all the
buffers which are in this GQueue. Taking care of converting them to rtx
This way, rtx packets are sent in the same order they have been requested.
(g_list_foreach traverse the queue from head to tail)
The GQueue is cleared between sending 2 master stream packets.
So for this GQueue to contain more than one element, it means that rtprtxsend
receives more than one rtx request between sending 2 master packets.
-- collision
When handling a GstRTPCollision event, if the ssrc is its rtx ssrc then
rtprtxsend clear its history and its pending retransmission queue.
Then it chooses a rtx_ssrc until it's different than master ssrc.
If the GstRTPCollision event does not contain its rtx ssrc, for example
its master ssrc or other, then it just forwards the event to upstream.
So that it can be handled by the rtppayloader.
rtprtxreceive element
-- basic mechanism
The same rtprtxreceive instance can receive several master streams and several
retransmission streams.
So it will try to dynamically associate a rtx ssrc with its master ssrc.
So that it can reconstruct the original from the proper rtx packet.
The algorithm is based on the fact that seqnums of different streams
(considering all master and all rtx streams) evolve at a different rate.
It means that the initial seqnum is random for each one and the offset could
also be different. So that they are statistically all different at a given
time. If bad luck then the association is delayed to the next rtx request.
The algorithm also needs to know if a given packet is a rtx packet or not.
To know this information there is the rtx-payload-types property. For now the
user as to configure it but later it will be automatically retreive this
information from SDP.
It needs to know if the current packet is rtx or not in order to know if
it can extract the OSN from the payload. Otherwise it would extract the OSN
even on master streams which means nothing and so it could do bad things.
In theory maybe it could work but we have this information in SDP so why not
using it to avoid bad associations.
Note that it also means that several master streams can have the same payload
type. And also several rtx streams can have the same payload type.
So the information from SDP which gives us which rtx payload type belong to
a give master payload type is not enough to do the association between rtx ssrc
and master ssrc.
rtprtxreceive works in SSRC-multiplexed mode, so it has one always sink and
src pad.
-- deeper in the association algorithm
When it receives a GstRTPRetransmissionRequest event it will remember the ssrc
and the seqnum from this request.
On incoming packets, if the packet has its ssrc already associated then it
knows if the ssrc is an rtx ssrc or a master stream ssrc.
If this is a rtx packet then it recontructs the original and pushs the result to
src pad as if it was a master packet.
If the ssrc is not yet associated rtprtxreceive checks the payload type.
if the packet has its payload type marked as rtx then it will extract the OSN
(original seqnum number) and lookup in its stored requests if a seqnum matchs.
If found, then it associates the current ssrc to the master ssrc marked in the
request. If not found it just drops the packet.
Then it removes the request from the stored requests.
If there are 2 requests with the same seqnum and different ssrc, then the
couple seqnum,ssrc is removed from the stored requests.
A stored request actually means that actually the couple seqnum,ssrc is stored.
If it's happens the request is droped but it avoids to do bad associations.
In this case the association is just delayed to the next request.
-- building original packet from rtx packet
Header, extensions, payload and padding are mostly the same. Except that the
OSN is removed from the payload. Then ssrc, seqnum, and original payload type
are correctly set. Original payload type is actually also stored when the
rtx request is handled.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment