dmabuf leak in Xwayland causing crash
When running srb2 (https://www.srb2.org/) version 2.2.1 under stock wine (wine-5.0 (Debian 5.0-)) under Xwayland (1.20.8) under sway git 45859be03f14fa0506ab8518feaec5ddb157e318, wlroots git cae533cad2b068dd13993a64fdc46c243c4984a2, eventually Xwayland will crash with a fatal error:
Fatal server error: wl_display@1 <clip> zwp_linux_buffer_params_v1_add@512: invalid arguments.
The reason for the invalid argument error is that eventually too many concurrent dmabuf file descriptors are open in the compositor. This is because the buffer handle returned from
zwp_linux_buffer_params_v1_create_immed() (called in
xwl_glamor_gbm_get_wl_buffer_for_pixmap()) is not released via
wl_buffer_destroy() (called from
xwl_glamor_gbm_destroy_pixmap()). I'm not sure which layer in the stack is responsible for releasing the server-side pixmap.
I was able to acquire the stack track from where
zwp_linux_buffer_params_v1_add() was being called that is causing the resource exhaustion:
Thread 1 "Xwayland" hit Breakpoint 1, zwp_linux_buffer_params_v1_add (modifier_lo=<optimized out>, modifier_hi=<optimized out>, stride=256, offset=9338880, plane_idx=1, fd=23, zwp_linux_buffer_params_v1=0x5561788b2640) at ./linux-dmabuf-unstable-v1-client-protocol.h:551 551 in ./linux-dmabuf-unstable-v1-client-protocol.h #0 zwp_linux_buffer_params_v1_add (modifier_lo=<optimized out>, modifier_hi=<optimized out>, stride=256, offset=9338880, plane_idx=1, fd=23, zwp_linux_buffer_params_v1=0x5561788b2640) at ./linux-dmabuf-unstable-v1-client-protocol.h:551 #1 xwl_glamor_gbm_get_wl_buffer_for_pixmap (pixmap=0x5561788a2640, created=<optimized out>) at ../../../../../hw/xwayland/xwayland-glamor-gbm.c:335 #2 0x0000556176968381 in xwl_present_flip (present_window=<optimized out>, crtc=<optimized out>, event_id=1, target_msc=2, pixmap=0x5561788a2640, sync_flip=1, damage=0x556178614a00) at ../../../../../hw/xwayland/xwayland-present.c:463 #3 0x0000556176a24745 in present_wnmd_flip (damage=0x556178614a00, sync_flip=<optimized out>, pixmap=<optimized out>, target_msc=<optimized out>, event_id=<optimized out>, crtc=<optimized out>, window=<optimized out>) at ../../../../present/present_wnmd.c:361 #4 present_wnmd_execute (vblank=vblank@entry=0x5561788b24a0, ust=3793823883, crtc_msc=1) at ../../../../present/present_wnmd.c:467 #5 0x0000556176a250b0 in present_wnmd_pixmap (window=<optimized out>, pixmap=0x5561788a2640, serial=1, valid=0x0, update=0x0, x_off=<optimized out>, y_off=0, target_crtc=<optimized out>, wait_fence=0x0, idle_fence=0x5561788a0260, options=8, window_msc=<optimized out>, divisor=0, remainder=0, notifies=0x0, num_notifies=0) at ../../../../present/present_wnmd.c:641 #6 0x0000556176a26761 in proc_present_pixmap (client=0x5561784e2a60) at ../../../../present/present_request.c:150 #7 0x0000556176a86614 in Dispatch () at ../../../../dix/dispatch.c:478 #8 0x0000556176a8a5e4 in dix_main (argc=10, argv=0x7ffd5319a5f8, envp=<optimized out>) at ../../../../dix/main.c:276 #9 0x00007f446dfb4e0b in __libc_start_main (main=0x55617695c250 <main>, argc=10, argv=0x7ffd5319a5f8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd5319a5e8) at ../csu/libc-start.c:308 #10 0x000055617695c28a in _start () at ../../../../present/present_request.c:346
Seems this has something to do with screen flipping at the end of drawing a frame but I don't know enough about X internals to exactly understand which module is responsible for cleaning up the pixmap.
If the root cause is due to an improper use of the X/GLX API, then at the very least, Xwayland should not crash.