video: AFD/Bar should maybe not be a meta
CC @boxerab, @ndufresne, @tpm, @nirbheek
Recently the AFD and Bar metas were merged into libgstvideo. I'm not sure if the information they convey should be in a meta, and as we're very very close to 1.16 release I would prefer to have them reverted for 1.16 unless my objections don't make any sense.
- Metas are generally for things that are changing often. This is not the case for AFD/Bar, it usually stays the same for quite some time.
- Muxers might need the information to write into the headers (MXF for example). Having it in a meta is suboptimal for this, especially as that also means that nothing is preventing the value to change all the time, which would not be supported by such muxers usually.
- With metas it is required that every single buffer has this attached, there must be no gaps. But AFD/Bar is something that generally applies to a longer period of time than a single frame. What would e.g. decklinkvideosink do if it sometimes gets buffers with AFD and sometimes without? Just not write it, take the last one if it saw one before, ...? It's unclear how to handle that cleanly and that should probably be at least documented how to handle that.
IMHO AFD/Bar and similar metadata shouldn't be a meta but either part of the caps, or become a separate sticky event.
Caps would cover all 3 points above but have the disadvantage (and advantage) that they affect negotiation, so things could potentially fail in muxers if AFD/Bar is changing (i.e. caps are changing) but the muxer does not support caps changes but also does not care at all about AFD/Bar (mp4mux for example). OTOH for muxers that support AFD/Bar but don't allow it to change (MXF) this is exactly the desired behaviour. A custom sticky event has the disadvantage that it might stay around longer than it should (dynamic stream switches where the 2nd stream has no information about such things).
I think we should take some time to design this properly first and not rush something into 1.16.