Skip to content

xserver
xserver
Merged
Opened by Olivier Fourdan@ofourdan
Report abuse

xwayland: do not crash if `glamor_set_pixmap_texture()` failed

If a client creates a very large pixmap (with a width of 32767), glamor_set_pixmap_texture() may fail because the corresponding FBO cannot be created.

Then, trying to upload content to that pixmap with no texture will crash Mesa, glamor and Xwayland:

  XXX fail to create fbo.
  (EE)
  (EE) Backtrace:
  (EE) 0: Xwayland (OsSigHandler+0x29)
  (EE) 1: libpthread.so.0 (funlockfile+0x50)
  (EE) 2: libc.so.6 (__memmove_avx_unaligned_erms+0x215)
  (EE) 3: dri/i965_dri.so (_mesa_format_convert+0xab3)
  (EE) 4: dri/i965_dri.so (_mesa_texstore+0x205)
  (EE) 5: dri/i965_dri.so (store_texsubimage+0x28c)
  (EE) 6: dri/i965_dri.so (intel_upload_tex+0x13b)
  (EE) 7: dri/i965_dri.so (texture_sub_image+0x134)
  (EE) 8: dri/i965_dri.so (texsubimage_err+0x150)
  (EE) 9: dri/i965_dri.so (_mesa_TexSubImage2D+0x48)
  (EE) 10: Xwayland (glamor_upload_boxes+0x246)
  (EE) 11: Xwayland (glamor_copy+0x4d1)
  (EE) 12: Xwayland (miCopyRegion+0x96)
  (EE) 13: Xwayland (miDoCopy+0x43c)
  (EE) 14: Xwayland (glamor_copy_area+0x24)
  (EE) 15: Xwayland (damageCopyArea+0xba)
  (EE) 16: Xwayland (compCopyWindow+0x31c)
  (EE) 17: Xwayland (damageCopyWindow+0xd3)
  (EE) 18: Xwayland (miResizeWindow+0x7b7)
  (EE) 19: Xwayland (compResizeWindow+0x3a)
  (EE) 20: Xwayland (ConfigureWindow+0xa96)
  (EE) 21: Xwayland (ProcConfigureWindow+0x7d)
  (EE) 22: Xwayland (Dispatch+0x320)
  (EE) 23: Xwayland (dix_main+0x366)
  (EE) 24: libc.so.6 (__libc_start_main+0xf3)
  (EE) 25: Xwayland (_start+0x2e)
  (EE)
  Fatal server error:
  (EE) Caught signal 11 (Segmentation fault). Server aborting
  (EE)

Unfortunately, glamor_set_pixmap_texture() does not return a status so the caller cannot tell whether it succeeded or failed, other than checking if the texture was set as expected.

Change Xwayland's GBM backend to check if the texture was set correctly when creating a pixmap from a buffer object and if that fails, fallback to the regular glamor pixmap creation.

Signed-off-by: Olivier Fourdan ofourdan@redhat.com Closes: #661 (closed)