kms: Only attempt to use AFBC compression when supported & fallback to not using AFBC when its unsupported
Rockchip RK3399 supports AFBC on some of its planes but only up to a resolution of 2560x1600 (pg 17 of RK3399 datasheet). I assume this limit is similar to other platforms.
The Kernel VOP driver doesn't actually have resolution limits for AFBC modifiers, so the output actually ends up mangled (from width=~2560px), as per the screenshot below:
Currently to workaround this, I run weston with WESTON_DISABLE_GBM_MODIFIERS=1
which is just a big hammer to completely disable AFBC (as well as other modifiers). This also means that AFBC is disabled even at resolutions when it should be supported.
To attempt to fix this, I have started a kernel branch which attempts to limit the resolution on AFBC planes. Running weston with this branch ends up with no output and weston handing because, well, atomic_check
returns an error. I think weston should fallback to not using AFBC when it is unsupported for a certain resolution?
How can we add some test & retry in weston, rather than giving up? I guess we need to do this format check when the plane is created (with a specific resolution) rather than erroring out far too late during atomic_check
?
I am happy to implement the code in kernel/weston for this, but I will need some guidance on what additional bits to poke.