Skip to content

x11 swapchain fails to check for DRI3 PixmapFromBuffers error

System information

Please post inxi -GSC -xx output (fenced with triple backticks) OR fill information below manually

System:    Host: cetoddle Kernel: 6.2.1-zen1 x86_64 bits: 64 compiler: gcc v: 12.2.0 Desktop: GNOME 43.3 
           tk: GTK 3.24.36 wm: gnome-shell dm: GDM Distro: NixOS 23.05 (Stoat) 
CPU:       Info: 10-Core model: 13th Gen Intel Core i5-13600K bits: 64 type: MT MCP arch: N/A rev: 1 cache: 
           L2: 24 MiB 
           flags: avx avx2 lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx bogomips: 139776 
           Speed: 1100 MHz min/max: 800/5100 MHz Core speeds (MHz): 1: 1100 2: 3500 3: 1100 4: 3500 5: 800 6: 3500 
           7: 800 8: 800 9: 3500 10: 800 11: 800 12: 800 13: 1100 14: 3500 15: 1100 16: 3500 17: 1100 18: 3500 
           19: 1100 20: 3500 
Graphics:  Message: No device data found. 
           Display: wayland server: X.Org 1.22.1.8 compositor: gnome-shell driver: loaded: N/A resolution: 
           1: 3840x2160~60Hz 2: 1080x1920~60Hz s-dpi: 96 
           OpenGL: renderer: NVIDIA GeForce RTX 3060/PCIe/SSE2 v: 4.6.0 NVIDIA 530.30.02 direct render: Yes

Describe the issue

Since nvidia driver release 525 trying to use the vulkan mesa driver under Xwayland gets into an error loop if Xwayland is using the nvidia gbm backend as output.

What is essentially happening is this:

  1. client initializes mesa vulkan driver
  2. vulkan driver tries to create swapchain and notices that the drm devices are not the same, configuring the prime setup - https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/src/vulkan/wsi/wsi_common_drm.c#L636
  3. vulkan driver creates DRM_FORMAT_MOD_LINEAR gbm buffer and sends it to Xwayland through DRI3.PixmapFromBuffers
  4. Xwayland tries to bind the buffer to a texture, but the nvidia driver does not allow the method it uses for linear buffers resulting in a GL error backtrace.
  5. Xwayland returns error BAD_ALLOC, but this is ignored by the mesa driver - https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/src/vulkan/wsi/wsi_common_x11.c#L2140
  6. At some later point mesa notices that the pixmap is invalid and tries to recreate the swapchain, restarting from step 2.

Essentially there are 2 bugs present:

  1. the prime setup does not check if DRM_FORMAT_MOD_LINEAR is an allowed modifier, it just always uses a linear buffer even if it isn't in the list of supported modifiers provided by the xcb server.
  2. failure to create a buffer on the xcb server side is ignored, resulting in an initialization loop.

In theory PRIME from intel to nvidia can work if all pieces support the OES_EGL_image_external extension, but since Xwayland does not it would be nice if the vulkan driver fails gracefully instead of getting stuck in a loop.

Steps to reproduce:

  1. Run Xwayland on nvidia GPU.
  2. Run VK_ICD_FILENAMES=<path_to_mesa_icd> vkcube
  3. Xwayland output is filled with xorg_backtrace()

Regression

This bug happened in older drivers as well, but it failed on Xwayland's side before reaching the GL code so it failed silently.

Any extra information would be greatly appreciated

More information about the bug can be found at xorg/xserver#1444 and https://github.com/Plagman/gamescope/issues/708

Comment from nvidia about DRM_FORMAT_MOD_LINEAR: https://github.com/NVIDIA/open-GPU-kernel-modules/discussions/243#discussioncomment-3283415

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information