Draft: nouveau: Rip out modifiers support
The Nouveau GL driver has claimed modifiers support for a couple of years now. However, it has always been a farse. Even though the modifier negotiation is probably fine and creating a surface with modifiers works, importing a surface with modifiers (a hard requirement for using modifiers with a compositor) is totally broken. There are two primary sources of breakage:
-
The old Nouveau kernel API requires that PTE kind and tile mode bits be set on BO creation and Nouveau relies on these bits in order to get correct PTE setup for tiled images. This is antithetical to the entire modifiers design. With modifiers, that sort of information is supposed to be communicated through the modifier, regardless of what is set on the BO. Without VM_BIND, there is no way for the GL driver to override the BO bits and use what the modifier says so it's dependent on the producer to set the BO metadata bits correctly. There is no fix for this without either re-defining the modifiers to require those bits to be set (which defeats the point of modifiers), extending the legacy kernel API in awkward ways to allow such an override, or converting GL to use VM_BIND.
-
The Nouveau GL driver ignores modifiers entirely when a dma-buf is imported. nvc0_resource_from_handle simply calls nv50_resoruce_from_handle which just pulls the tile mode out of the BO and blindly smashes it into the miptree. Modifiers appear nowhere in the function. This means that, even if we could override the tile mode and PTE kind set on the BO from the modifier, we aren't doing so and never have.
Between these two things, modifiers basically exist to let us do tiled scan-out. The kernel, as far as Dave can tell, does handle modifiers correctly. However, Everything involving client <-> compositor communication is relying entirely on the BO side-band and ignoring modifiers.
There's no good way to fix any of this in a way that's back-portable. The only back-portable thing we can do is rip it all out.