[vmwgfx] driver shouldn't crash badly when memory starved
System information
System:
Host: pc Kernel: 5.8.8-arch1-1 x86_64 bits: 64 compiler: N/A
Desktop: GNOME 3.36.6 tk: GTK 3.24.23 wm: gnome-shell dm: GDM
Distro: Arch Linux
CPU:
Info: 2x Single Core model: Intel Core i7-4770K bits: 64 type: SMP
arch: Haswell rev: 3 L2 cache: 16.0 MiB
flags: avx avx2 lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3
bogomips: 14005
Speed: 3500 MHz min/max: N/A Core speeds (MHz): 1: 3500 2: 3500
Graphics:
Device-1: VMware SVGA II Adapter driver: vmwgfx v: 2.18.0.0
bus ID: 00:0f.0 chip ID: 15ad:0405
Display: x11 server: X.Org 1.20.9 compositor: gnome-shell driver: vmware
unloaded: fbdev,modesetting,vesa s-res: 1387x773 s-dpi: 96
OpenGL: renderer: SVGA3D; build v: 3.3 Mesa 20.3.0-devel (git-4868ce1451)
direct render: Yes
Describe the issue
When logging into GNOME with a high-res background and only 256MB of vram, the process crash in a loop. Journalctl shows gnome-shell: VMware: vmw_ioctl_command error Cannot allocate memory.
Increasing VRAM, or using an empty background, solves this but not disabling 3D acceleration altogether.
I get that somehow the driver shouldn't babysit every kind of possible edge case, but putting aside that I don't see why you couldn't fallback to RAM on exhaustion, it should at least fail in a graceful enough way not to bring down the whole process.
Log files as attachment
#0 0x00007fda33fc8615 in raise () at /usr/lib/libc.so.6
#1 0x00005603f368bad9 in ()
#2 0x00007fda33fc86a0 in <signal handler called> () at /usr/lib/libc.so.6
#3 0x00007fda33fc8615 in raise () at /usr/lib/libc.so.6
#4 0x00007fda33fb1862 in abort () at /usr/lib/libc.so.6
#5 0x00007fda173b2d94 in vmw_ioctl_command
(throttle_us=0, flags=<optimized out>, imported_fence_fd=<optimized out>, pfence=0x7fffd324b4d8, size=<optimized out>, commands=0x5603f56bcf18, cid=<optimized out>, vws=0x5603f56860d0) at ../mesa/src/util/list.h:61
#6 vmw_swc_flush(svga_winsys_context*, pipe_fence_handle**) (swc=0x5603f56bce30, pfence=0x7fffd324b5a8)
at ../mesa/src/gallium/winsys/svga/drm/vmw_context.c:239
#7 0x00007fda1736ec74 in svga_context_flush(svga_context*, pipe_fence_handle**)
(svga=0x5603f562a470, pfence=pfence@entry=0x0) at ../mesa/src/gallium/drivers/svga/svga_context.c:397
#8 0x00007fda1737c5df in svga_hwtnl_prim(svga_hwtnl*, SVGA3dPrimitiveRange const*, unsigned int, unsigned int, unsigned int, pipe_resource*, unsigned int, unsigned int, pipe_draw_indirect_info const*, pipe_stream_output_target const*)
(hwtnl=0x5603f56504b0, range=0x7fffd324b6c0, vcount=4, min_index=<optimized out>, max_index=<optimized out>, ib=0x0, start_instance=0, instance_count=1, indirect=0x0, so_vertex_count=0x0)
at ../mesa/src/gallium/drivers/svga/svga_draw.c:1161
#9 0x00007fda173b1285 in simple_draw_arrays
(vertices_per_patch=3 '\003', instance_count=1, start_instance=0, count=4, start=0, prim=<optimized out>, hwtnl=0x5603f56504b0) at ../mesa/src/gallium/drivers/svga/svga_draw_arrays.c:202
#10 svga_hwtnl_draw_arrays(svga_hwtnl*, pipe_prim_type, unsigned int, unsigned int, unsigned int, unsigned int, ubyte)
(hwtnl=0x5603f56504b0, prim=<optimized out>, start=0, count=4, start_instance=0, instance_count=1, vertices_per_patch=3 '\003') at ../mesa/src/gallium/drivers/svga/svga_draw_arrays.c:278
#11 0x00007fda1737eb7e in retry_draw_arrays (vertices_per_patch=3 '\003', instance_count=1, start_instance=0, count=4, start=0, prim=PIPE_PRIM_TRIANGLE_FAN, svga=0x5603f562a470) at ../mesa/src/gallium/drivers/svga/svga_pipe_draw.c:60
#12 svga_draw_vbo(pipe_context*, pipe_draw_info const*) (pipe=0x5603f562a470, info=<optimized out>) at ../mesa/src/gallium/drivers/svga/svga_pipe_draw.c:355
#13 0x00007fda16abcb08 in cso_draw_vbo (info=0x7fffd324b840, cso=<optimized out>) at ../mesa/src/gallium/auxiliary/cso_cache/cso_context.c:1736
#14 st_draw_vbo(gl_context*, _mesa_prim const*, GLuint, _mesa_index_buffer const*, GLboolean, GLuint, GLuint, GLuint, GLuint, gl_transform_feedback_object*, unsigned int) (ctx=0x7fffd324b840, prims=0x7fffd324b900, nr_prims=<optimized out>, ib=0x0, index_bounds_valid=<optimized out>, min_index=<optimized out>, max_index=<optimized out>, num_instances=1, base_instance=0, tfb_vertcount=0x0, stream=0) at ../mesa/src/mesa/state_tracker/st_draw.c:266
#15 0x00007fda1773e174 in _mesa_draw_arrays (drawID=0, baseInstance=<optimized out>, numInstances=<optimized out>, count=<optimized out>, start=<optimized out>, mode=<optimized out>, ctx=<optimized out>) at ../mesa/src/mesa/main/draw.c:367
#16 _mesa_draw_arrays.constprop.0 (ctx=<optimized out>, mode=<optimized out>, start=<optimized out>, count=<optimized out>, numInstances=<optimized out>, baseInstance=<optimized out>, drawID=<optimized out>) at ../mesa/src/mesa/main/draw.c:348
#17 0x00007fda33c5aedf in _cogl_framebuffer_draw_attributes (flags=(COGL_DRAW_SKIP_JOURNAL_FLUSH | COGL_DRAW_SKIP_PIPELINE_VALIDATION | COGL_DRAW_SKIP_FRAMEBUFFER_FLUSH), n_attributes=<optimized out>, attributes=0x5603f815acb0, n_vertices=4, first_vertex=<optimized out>, mode=COGL_VERTICES_MODE_TRIANGLE_FAN, pipeline=<optimized out>, framebuffer=0x5603f86f6590) at ../mutter/cogl/cogl/cogl-framebuffer.c:2091
#18 _cogl_journal_flush_modelview_and_entries (batch_start=<optimized out>, batch_len=1, data=data@entry=0x7fffd324bbc0) at ../mutter/cogl/cogl/cogl-journal.c:321
#19 0x00007fda33c5b01d in _cogl_journal_flush_pipeline_and_entries (batch_start=<optimized out>, batch_len=batch_len@entry=1, data=data@entry=0x7fffd324bbc0) at ../mutter/cogl/cogl/cogl-journal.c:427
#20 0x00007fda33c5b441 in batch_and_call (data=<optimized out>, batch_callback=<optimized out>, can_batch_callback=<optimized out>, n_entries=<optimized out>, entries=<optimized out>) at ../mutter/cogl/cogl/cogl-journal.c:445
#21 batch_and_call (data=0x7fffd324bbc0, batch_callback=0x7fda33c5af60 <_cogl_journal_flush_pipeline_and_entries>, can_batch_callback=<optimized out>, n_entries=<optimized out>, entries=<optimized out>) at ../mutter/cogl/cogl/cogl-journal.c:238
#22 _cogl_journal_flush_texcoord_vbo_offsets_and_entries (batch_start=<optimized out>, batch_len=<optimized out>, data=data@entry=0x7fffd324bbc0) at ../mutter/cogl/cogl/cogl-journal.c:544
#23 0x00007fda33c5b74a in batch_and_call (data=<optimized out>, batch_callback=<optimized out>, can_batch_callback=<optimized out>, n_entries=<optimized out>, entries=<optimized out>) at ../mutter/cogl/cogl/cogl-journal.c:269
#24 batch_and_call (data=0x7fffd324bbc0, batch_callback=0x7fda33c5b200 <_cogl_journal_flush_texcoord_vbo_offsets_and_entries>, can_batch_callback=<optimized out>, n_entries=2, entries=<optimized out>) at ../mutter/cogl/cogl/cogl-journal.c:238
#25 _cogl_journal_flush_vbo_offsets_and_entries (batch_start=<optimized out>, batch_len=2, data=<optimized out>) at ../mutter/cogl/cogl/cogl-journal.c:651
#26 0x00007fda33c5bc99 in batch_and_call (data=<optimized out>, batch_callback=<optimized out>, can_batch_callback=<optimized out>, n_entries=<optimized out>, entries=<optimized out>) at ../mutter/cogl/cogl/cogl-journal.c:269
#27 batch_and_call (data=<optimized out>, batch_callback=0x7fda33c5b4f0 <_cogl_journal_flush_vbo_offsets_and_entries>, can_batch_callback=<optimized out>, n_entries=<optimized out>, entries=<optimized out>) at ../mutter/cogl/cogl/cogl-journal.c:238
Any extra information would be greatly appreciated
Intriguingly enough, when updating from 20.1.7 to latest git, running the WebGL conformance suite stopped to cause the same kind of issue (it necessarily required a whole 2GB of VRAM, now 256MB is already enough).