Skip to content

xwayland: do not crash if `glamor_set_pixmap_texture()` failed

Olivier Fourdan requested to merge ofourdan/xserver:issue661 into master

If a client creates a very large pixmap (with a width of 32767), glamor_set_pixmap_texture() may fail because the corresponding FBO cannot be created.

Then, trying to upload content to that pixmap with no texture will crash Mesa, glamor and Xwayland:

  XXX fail to create fbo.
  (EE)
  (EE) Backtrace:
  (EE) 0: Xwayland (OsSigHandler+0x29)
  (EE) 1: libpthread.so.0 (funlockfile+0x50)
  (EE) 2: libc.so.6 (__memmove_avx_unaligned_erms+0x215)
  (EE) 3: dri/i965_dri.so (_mesa_format_convert+0xab3)
  (EE) 4: dri/i965_dri.so (_mesa_texstore+0x205)
  (EE) 5: dri/i965_dri.so (store_texsubimage+0x28c)
  (EE) 6: dri/i965_dri.so (intel_upload_tex+0x13b)
  (EE) 7: dri/i965_dri.so (texture_sub_image+0x134)
  (EE) 8: dri/i965_dri.so (texsubimage_err+0x150)
  (EE) 9: dri/i965_dri.so (_mesa_TexSubImage2D+0x48)
  (EE) 10: Xwayland (glamor_upload_boxes+0x246)
  (EE) 11: Xwayland (glamor_copy+0x4d1)
  (EE) 12: Xwayland (miCopyRegion+0x96)
  (EE) 13: Xwayland (miDoCopy+0x43c)
  (EE) 14: Xwayland (glamor_copy_area+0x24)
  (EE) 15: Xwayland (damageCopyArea+0xba)
  (EE) 16: Xwayland (compCopyWindow+0x31c)
  (EE) 17: Xwayland (damageCopyWindow+0xd3)
  (EE) 18: Xwayland (miResizeWindow+0x7b7)
  (EE) 19: Xwayland (compResizeWindow+0x3a)
  (EE) 20: Xwayland (ConfigureWindow+0xa96)
  (EE) 21: Xwayland (ProcConfigureWindow+0x7d)
  (EE) 22: Xwayland (Dispatch+0x320)
  (EE) 23: Xwayland (dix_main+0x366)
  (EE) 24: libc.so.6 (__libc_start_main+0xf3)
  (EE) 25: Xwayland (_start+0x2e)
  (EE)
  Fatal server error:
  (EE) Caught signal 11 (Segmentation fault). Server aborting
  (EE)

Unfortunately, glamor_set_pixmap_texture() does not return a status so the caller cannot tell whether it succeeded or failed, other than checking if the texture was set as expected.

Change Xwayland's GBM backend to check if the texture was set correctly when creating a pixmap from a buffer object and if that fails, fallback to the regular glamor pixmap creation.

Signed-off-by: Olivier Fourdan ofourdan@redhat.com Closes: #661 (closed)

Merge request reports