GitLab

From similar downstream reports, in some random cases, Xwayland crashes in glamor_get_pixmap_texture():

Truncated backtrace:
0: /usr/bin/Xwayland (OsLookupColor+0x13d) [0x55da62a5d31d]
1: /lib64/libpthread.so.0 (funlockfile+0x50) [0x7f7cfd1c307f]
2: /usr/bin/Xwayland (glamor_get_pixmap_texture+0x81) [0x55da62908061]
3: /usr/bin/Xwayland (glamor_create_gc+0x70e8) [0x55da62919178]
4: /usr/bin/Xwayland (glamor_create_gc+0x751c) [0x55da6291991c]
5: /usr/bin/Xwayland (glamor_create_gc+0x7b2e) [0x55da6291a0be]
6: /usr/bin/Xwayland (glamor_create_gc+0x9b2c) [0x55da6291dd1c]
7: /usr/bin/Xwayland (DamageRegionAppend+0x6af) [0x55da629cb04f]
8: /usr/bin/Xwayland (glamor_create_gc+0x10a22) [0x55da6292c1f2]
9: /usr/bin/Xwayland (AddTraps+0x4275) [0x55da629c0955]
10: /usr/bin/Xwayland (SendErrorToClient+0x35e) [0x55da62a2735e]
11: /usr/bin/Xwayland (InitFonts+0x3c6) [0x55da62a2b396]
12: /lib64/libc.so.6 (__libc_start_main+0xf3) [0x7f7cfd00c413]
13: /usr/bin/Xwayland (_start+0x2e) [0x55da628f937e]

Prior to this crash, the logs show:

 (WW) glamor: Expect reduced performance.
 (WW) glamor: Failed to allocate 1922x1175 FBO due to GL_OUT_OF_MEMORY.
 (EE) glamor0: GL error: GL_OUT_OF_MEMORY in glTexSubImage

Considering glamor_get_pixmap_texture() is a pretty simple function, the segfault occurs most likely because pixmap_priv->fbo is NULL.

The error messages come from _glamor_create_tex() which is called by glamor_create_fbo(), and when this occurs, pixmap_priv->fbo is set to NULL because, well, the FBO could not be allocated.

Downstream bug: https://bugzilla.redhat.com/show_bug.cgi?id=1648475