Skip to content

rtpulpfec: Not usable as a generic FEC mechanism

This is a result of how ULPFEC works and how GStreamer works.

The problem here is that ULPFEC uses the same seqnum space for FEC packets as for non-FEC packets, so whenever you lost a packet you don't know if it was a FEC packet (uninteresting) or a non-FEC packet (problematic, needs to be recovered somehow).

This means that when rtpjitterbuffer notices a missing packet (FEC or not), it will send a lost packet event and will also mark the next buffer as a discontinuity. The lost packet event is already ignored in this case by the depayloader (it has a "might be FEC" flag).

The discont flag on the other hand will cause the following depayloader to drop any data it has currently queued up (e.g. to rebuild a fragmented frame from multiple packets), and also causes the next output to have a discont flag so this goes up to the sinks and e.g. baseaudiosink will resync its ringbuffer then (inserting silence or dropping samples as needed).

This means that with a higher FEC percentage (say 100%), you can get a theoretically higher recovery rate but nonetheless bad user experience due to data being dropped and resyncing because of the discont flag. Every lost FEC packet specifically will cause a discont flag, so the more FEC packets you have the more likely it is that one is lost and the more likely you drop data unnecessarily and cause resyncing.

I don't know how the ULPFEC designers envisioned this to be used, but I don't think it is possible to use ULPFEC as a generic FEC mechanism without further mechanisms to mitigate the above problem.

In my application I'm now adding another sequence number in an RTP header extension to distinguish between actually missing packets or only missing FEC packets or otherwise irrelevant packet loss (because recovered for example), and set the discont flag on buffers according to this second layer of sequence numbers.

Pexip (CC @hgr @fludkov) implemented a similar mechanism in the VP8/VP9 depayloaders based on actual bitstream parsing and taking: https://github.com/pexip/gst-plugins-good/commits/master

There are multiple things to do here

  • Implement a non-stupid FEC mechanism like flexfec or whatever, and discourage usage of ULPFEC
  • Document the uselessness of ULPFEC more visible so that people don't just try using it and then are disappointed that it does not improve anything or even worsens the situation
  • Merge the Pexip patches in one form or another for making ULPFEC work at least for VP8/VP9
  • Check if similar work can also be done for other codecs (or was done already? @hgr @fludkov)