Implement vulkan renderer
Reworked version of #1214, building upon the new wlr_allocator abstraction. A lot of the old discussion points have been resolved.
- the renderer has a hard dependency on VK_EXT_image_drm_format_modifier. No undefined image importing hackery or linear image layouts anymore
- the old wlr_render_surface concept is removed, vulkan integrates very well with the wlr_allocator concept :)
- this does no API changes to wlroots at all
- The vulkan renderer is an optional compile-time feature. Only if compiled with vulkan and if the WLR_RENDERER=vulkan environment variable is set, the vulkan renderer will be created by wlr_renderer_autocreate.
- A couple of functions are currently not implemented:
wlr_renderer_read_pixels
,(functions have been removed from wlroots). We can implement all of them but implementations would be poor, I would much rather propose interface changes to implement them efficiently, without completely blocking the compositor. These functions being unimplemented should only have an impact on multi-GPU support and wlr_screencopy_v1 not working under vulkan.wlr_renderer_blit_dmabuf
,wlr_texture_to_dmabuf
- There are still a couple of smaller improvements noted in the code but nothing of this seems too important to me, it's mostly performance improvements or simplifications.
- This support YCbCr textures (for shm as well as dmabuf texture importing) but I couldn't find a way to test that yet. It does not support rendering to such formats but I don't think that's ever needed.
As this isn't perfect and needs a lot of testing on all possible hardware, the renderer currently always enables the vulkan validation layers. You will need this patch as there were errors in the validation layers for VK_EXT_image_drm_format_modifier. You will need to build the vulkan validation layers from master as there was an error with validation of a used extension just fixed recently.
I tested this on an intel 8th gen GPU with the anv mesa MR (there isn't support for importing YCbCr dmabufs yet), using tinywl with the wayland and drm backend, running the sample weston-* applications as well as firefox and some GTK applications. Sadly, I don't have a >=GFX9 (vega) AMD GPU, and since radv already has support for the vulkan extension on those GPUs I'd be thankful for people testing it. Tests on any other driver/platform where the extension is available would certainly be useful as well.
If you can think of applications that this should be tested with, let me know (something that uses YCbCr shm buffers would be nice).