Follow-up from "codecs: av1dec: Enable scalable decoding"
The following discussion from !2475 (merged) should be addressed:
This patch can really solve the multi layer problem, the correctness is OK, but have some performance flaw because of the caps negotiation.
For example, there are 2 spatial layer, 0 with 800x600 res and 1 with 1092x1080 res. According to spec, we should only output the higher layer by default, which is the 1 with 1920x1080 res. But we need to decode both layers(The higher layer needs to reference the lower layer). The coded picture from both layers come in interweave sequence, and in the new_picture(), because the resolution changes, we need to re-negotiate the caps with the downstream element again and again. The is a big waste and have performance problem.
If we want to avoid this, we should notify the sub-class whether this picture will be output or not when call the new_picture(). And the sub-class should maintain a buffer_pool for each non-output layers internally, and allocate the picutre/buffers/surfaces from that internal buffer pool to avoid that re-negotiation.
This may take a big effect(Each internal layer can also change their resolution dynamically). I think this patch can already make us conform to the SPEC and pass all the conformance test about his. This multi layers feature is more or less like the SVC feature of the HEVC, we are not sure whether it will be widely used. And I think we can add a TODO for that, and this patch is already OK for me.