Xwayland: invalid operation in nvidia gbm import

related github issue: https://github.com/Plagman/gamescope/issues/708

When Xwayland is run on a nvidia gbm-backed egldisplay, if the application uses a different gbm backend it causes a failure during initialization of the framebuffers. Since this is an operation that can spuriously fail most clients will proceed to retry, resulting in the following error being spammed:

(EE) glamor0: GL error: GL_INVALID_OPERATION error generated. <image> and <target> are incompatible
(EE) 
(EE) Backtrace:
(EE) unw_get_proc_name failed: no unwind info found [-10]
(EE) 0: /usr/lib/libnvidia-eglcore.so.525.85.05 (?+0x0) [0x620962b07037]
(EE) unw_get_proc_name failed: no unwind info found [-10]
(EE) 1: /usr/lib/libnvidia-eglcore.so.525.85.05 (?+0x0) [0x620962b071b3]
(EE) unw_get_proc_name failed: no unwind info found [-10]
(EE) 2: /usr/lib/libnvidia-eglcore.so.525.85.05 (?+0x0) [0x620962b0739a]
(EE) unw_get_proc_name failed: no unwind info found [-10]
(EE) 3: /usr/lib/libnvidia-eglcore.so.525.85.05 (?+0x0) [0x620962bede0a]
(EE) 4: Xwayland (xwl_glamor_gbm_create_pixmap_for_bo+0x1d3) [0xc6c5a672613]
(EE) 5: Xwayland (xwl_glamor_gbm_create_pixmap+0x152) [0xc6c5a672a62]
(EE) 6: Xwayland (ProcCreatePixmap+0x12e) [0xc6c5a6c75ee]
(EE) 7: Xwayland (Dispatch+0xce8) [0xc6c5a6cfcc8]
(EE) 8: Xwayland (main+0x182e) [0xc6c5a65c23e]
(EE) 9: /usr/lib/libc.so.6 (__libc_init_first+0x90) [0x620968161290]
(EE) 10: /usr/lib/libc.so.6 (__libc_start_main+0x8a) [0x62096816134a]
(EE) 11: Xwayland (_start+0x25) [0xc6c5a65da85]
(EE) 
XXX fail to create fbo.

I've been able to reliably reproduce this issue by running Xwayland on the 525.89.02 nvidia driver and then running VK_ICD_FILENAMES=<path_to_icds>/intel_icd.json vkcube. This does require an nvidia dGPU, an intel iGPU and both drivers must have KMS enabled so there are 2 drm drivers.

---

Finally here is my comment about what I believe the issue is: ¹

After some more digging I believe I've found the root cause of this issue:

1. Since driver 525 nvidia added support for the vendor-neutral DRM_FORMAT_MOD_LINEAR format modifier², but it is marked as external_only³ meaning it can only be loaded into GL_TEXTURE_EXTERNAL_OES⁴.
2. Xwayland reads the list of supported modifiers and sends it to the client/application, ignoring external_only
3. The client initializes a different graphics backend for some reason. This backend does not support any of the nvidia-specific buffer modifiers, but it does support DRM_FORMAT_MOD_LINEAR.
4. Since it is supported it creates this buffer and sends it back to Xwayland to finish initialization of the output buffers.
5. Xwayland then proceeds to import the buffer using GL_TEXTURE_2D⁵ which is incompatible with external_only buffer formats, leading to the error we're seeing.

In my case a vulkan loader bug caused it to initialize the wrong driver,⁶ but it is reasonable to assume any case where the application running under Xwayland tries to use a different GPU (which supported DRM_FORMAT_MOD_LINEAR) will run into this issue.

As for fixing it Xwayland needs to either filter out any modifiers marked as external_only or add support for GL_TEXTURE_EXTERNAL_OES which appears to be quite an extensive change

1. https://github.com/Plagman/gamescope/issues/708#issuecomment-1436051840 ↩

2. https://github.com/NVIDIA/open-GPU-kernel-modules/discussions/243#discussioncomment-3283415 ↩

3. https://registry.khronos.org/EGL/extensions/EXT/EGL_EXT_image_dma_buf_import_modifiers.txt ↩

4. https://registry.khronos.org/OpenGL/extensions/OES/OES_EGL_image_external.txt ↩

5. https://registry.khronos.org/OpenGL/extensions/OES/OES_EGL_image.txt ↩

6. To determine if this is the cause set VK_LOADER_DEBUG=driver and run the application. It should first print vkDeviceCreate using the nvidia driver (this is gamescope creating the window), then print vkDeviceCreate with a different driver when the application initializes vulkan. ↩