Don't crash when libd3d12.so can't be found (!20145) · Merge requests · Mesa / mesa

Jan Alexander Steffens requested to merge heftig/mesa:d3d12-crash into main Dec 04, 2022

On Arch Linux testing (Mesa 22.3.0), running Xephyr :3 crashes with a segfault and the following backtrace:

Backtrace

#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007f01538876b3 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007f0153837958 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007f015382153d in __GI_abort () at abort.c:79
#4  0x000055b5b53f5420 in OsAbort () at ../xorg-server-21.1.4/os/utils.c:1352
#5  0x000055b5b5401ccc in AbortServer () at ../xorg-server-21.1.4/os/log.c:879
#6  FatalError (f=<optimized out>) at ../xorg-server-21.1.4/os/log.c:1017
#7  0x000055b5b53f2281 in OsSigHandler (unused=<optimized out>, sip=<optimized out>, signo=11) at ../xorg-server-21.1.4/os/osinit.c:156
#8  OsSigHandler (signo=11, sip=<optimized out>, unused=<optimized out>) at ../xorg-server-21.1.4/os/osinit.c:110
#9  <signal handler called>
#10 _dl_close (_map=0x0) at dl-close.c:795
#11 0x00007f015394ee3e in __GI__dl_catch_exception (exception=exception@entry=0x7ffe48901230, operate=<optimized out>, args=<optimized out>) at /usr/src/debug/glibc/elf/dl-error-skeleton.c:208
#12 0x00007f015394eef3 in __GI__dl_catch_error (objname=0x7ffe48901288, errstring=0x7ffe48901290, mallocedp=0x7ffe48901287, operate=<optimized out>, args=<optimized out>) at /usr/src/debug/glibc/elf/dl-error-skeleton.c:227
#13 0x00007f01538814af in _dlerror_run (operate=<optimized out>, args=<optimized out>) at dlerror.c:138
#14 0x00007f0153881206 in __dlclose (handle=<optimized out>) at dlclose.c:31
#15 0x00007f0151d1ef4c in d3d12_destroy_screen () at ../mesa-22.3.0/src/gallium/drivers/d3d12/d3d12_screen.cpp:744
#16 0x00007f0151d1e159 in d3d12_create_dxcore_screen () at ../mesa-22.3.0/src/gallium/drivers/d3d12/d3d12_dxcore_screen.cpp:236
#17 0x00007f01512a8324 in sw_screen_create_named () at ../mesa-22.3.0/src/gallium/auxiliary/target-helpers/sw_helper.h:70
#18 sw_screen_create_vk () at ../mesa-22.3.0/src/gallium/auxiliary/target-helpers/sw_helper.h:102
#19 0x00007f01518d163a in pipe_loader_sw_create_screen () at ../mesa-22.3.0/src/gallium/auxiliary/pipe-loader/pipe_loader_sw.c:425
#20 0x00007f01518d158a in pipe_loader_create_screen_vk () at ../mesa-22.3.0/src/gallium/auxiliary/pipe-loader/pipe_loader.c:171
#21 0x00007f01512a8c1b in drisw_init_screen () at ../mesa-22.3.0/src/gallium/frontends/dri/drisw.c:578
#22 0x00007f01512b295f in driCreateNewScreen2 () at ../mesa-22.3.0/src/gallium/frontends/dri/dri_util.c:143
#23 0x000055b5b54045a5 in __glXDRIscreenProbe (pScreen=0x55b5b6d07eb0) at ../xorg-server-21.1.4/glx/glxdriswrast.c:448
#24 0x000055b5b54031a7 in xorgGlxServerInit (pcbl=<optimized out>, param=<optimized out>, ext=<optimized out>) at ../xorg-server-21.1.4/glx/glxext.c:550
#25 xorgGlxServerInit (pcbl=<optimized out>, param=<optimized out>, ext=<optimized out>) at ../xorg-server-21.1.4/glx/glxext.c:525
#26 0x000055b5b5346fec in _CallCallbacks (pcbl=0x55b5b54be7e0 <vndInitCallbackListPtr.lto_priv.0>, call_data=0x55b5b6d1f820) at ../xorg-server-21.1.4/dix/dixutils.c:743
#27 0x000055b5b54257c1 in CallCallbacks (call_data=0x55b5b6d1f820, pcbl=<optimized out>) at ../xorg-server-21.1.4/include/callback.h:83
#28 GlxExtensionInit () at ../xorg-server-21.1.4/glx/vndext.c:244
#29 GlxExtensionInit () at ../xorg-server-21.1.4/glx/vndext.c:212
#30 0x000055b5b5348e62 in InitExtensions (argc=2, argv=0x7ffe48902248) at ../xorg-server-21.1.4/mi/miinitext.c:272
#31 dix_main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at ../xorg-server-21.1.4/dix/main.c:194
#32 0x00007f0153822290 in __libc_start_call_main (main=main@entry=0x55b5b52c8880 <main>, argc=argc@entry=2, argv=argv@entry=0x7ffe48902248) at ../sysdeps/nptl/libc_start_call_main.h:58
#33 0x00007f015382234a in __libc_start_main_impl (main=0x55b5b52c8880 <main>, argc=2, argv=0x7ffe48902248, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffe48902238) at ../csu/libc-start.c:381
#34 0x000055b5b52c88e5 in _start () at ../sysdeps/x86_64/start.S:115

It attempts to initialize the d3d12 driver, which fails because libd3d12.so does not exist. Destroying the partly-initialized screen then runs dlclose on a NULL pointer, leading to a crash.

The following commit fixes the issue:

d3d12: Don't crash when libd3d12.so can't be found
d3d12_destroy_screen is called by d3d12_create_dxcore_screen after d3d12_init_screen_base fails and attempts to call util_dl_close on a NULL pointer, leading to an abort.

To fix this, only close the library after if it was actually opened.

Looking through the code I also found a potential similar crash in dzn, leading to a second commit:

dzn: Don't crash when libd3d12.so can't be found
dzn_instance_create will call dzn_instance_destroy when the d3d12 library fails to load. Just like the issue in d3d12_screen, this will lead to a crash because d3d12_mod is NULL.

To fix this, only close the library after if it was actually opened.

Edited Dec 04, 2022 by Jan Alexander Steffens

Don't crash when libd3d12.so can't be found

Merge request reports