COMPOSITE + ROOTLESS are incompatible
If COMPOSITE and ROOTLESS are both enabled in XQuartz, we can get a double free of a pixmap. The first free actually makes it available for scratch and the second frees it. Another allocation can come along in the future and get the deallocated scratch space and boom. Trivailly reproducable with ASAan enabled and using nedit. Just open a menu and then another menu.
[603505.833] GetScratchPixmapHeader: pScreen->pScratchPixmap = 0x124039560
[603505.833] GetScratchPixmapHeader: pPixmap = 0x124039560
[603505.833] (EE)
[603505.833] (EE) Backtrace:
[603505.833] (EE) 0: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (xorg_backtrace+0x190) [0x10479ab6c]
[603505.833] (EE) 1: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (GetScratchPixmapHeader+0x154) [0x1042fb6fc]
[603505.833] (EE) 2: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (RootlessStartDrawing+0x488) [0x104057b7c]
[603505.834] (EE) 3: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (miPaintWindow+0x1c8) [0x104156f9c]
[603505.834] (EE) 4: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (miWindowExposures+0x358) [0x104159968]
[603505.834] (EE) 5: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (miHandleValidateExposures+0x138) [0x1041a9c38]
[603505.834] (EE) 6: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (UnmapWindow+0x954) [0x10434eab8]
[603505.834] (EE) 7: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (ProcUnmapWindow+0x2a0) [0x104213ce8]
[603505.835] (EE) 8: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (Dispatch+0x8bc) [0x10420b910]
[603505.835] (EE) 9: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (dix_main+0x11a4) [0x10425c7d4]
[603505.835] (EE) 10: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (server_thread+0x1e8) [0x10404cc14]
[603505.835] (EE) 11: /usr/lib/system/libsystem_pthread.dylib (_pthread_start+0x94) [0x18599506c]
[603505.835] (EE) 12: /usr/lib/system/libsystem_pthread.dylib (thread_start+0x8) [0x18598fe2c]
[603505.840] compReallocPixmap: pOld: 0x124039560, pNew: 0x128338800
[603505.840] (EE)
[603505.840] (EE) Backtrace:
[603505.841] (EE) 0: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (xorg_backtrace+0x190) [0x10479ab6c]
[603505.841] (EE) 1: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (compReallocPixmap+0x508) [0x1047ea60c]
[603505.841] (EE) 2: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (compConfigNotify+0x444) [0x1048018dc]
[603505.841] (EE) 3: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (present_config_notify+0x1c0) [0x104496738]
[603505.841] (EE) 4: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (ConfigureWindow+0x2394) [0x1043591a0]
[603505.842] (EE) 5: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (ProcConfigureWindow+0x40c) [0x104214500]
[603505.842] (EE) 6: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (Dispatch+0x8bc) [0x10420b910]
[603505.842] (EE) 7: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (dix_main+0x11a4) [0x10425c7d4]
[603505.842] (EE) 8: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (server_thread+0x1e8) [0x10404cc14]
[603505.842] (EE) 9: /usr/lib/system/libsystem_pthread.dylib (_pthread_start+0x94) [0x18599506c]
[603505.842] (EE) 10: /usr/lib/system/libsystem_pthread.dylib (thread_start+0x8) [0x18598fe2c]
[603505.843] FreeScratchPixmapHeader: pScreen->pScratchPixmap = 0x0, pPixmap = 0x124039560
[603505.843] (EE)
[603505.843] (EE) Backtrace:
[603505.843] (EE) 0: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (xorg_backtrace+0x190) [0x10479ab6c]
[603505.843] (EE) 1: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (FreeScratchPixmapHeader+0xd0) [0x1042fb8d8]
[603505.843] (EE) 2: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (RootlessStopDrawing+0x228) [0x1040585b4]
[603505.843] (EE) 3: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (RootlessRedisplay+0x2c) [0x10405a384]
[603505.843] (EE) 4: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (StartFrameResize+0x520) [0x104096c54]
[603505.844] (EE) 5: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (RootlessResizeWindow+0x4b8) [0x10409aebc]
[603505.844] (EE) 6: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (compResizeWindow+0x160) [0x1047fd2a8]
[603505.844] (EE) 7: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (ConfigureWindow+0x33d0) [0x10435a1dc]
[603505.844] (EE) 8: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (ProcConfigureWindow+0x40c) [0x104214500]
[603505.844] (EE) 9: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (Dispatch+0x8bc) [0x10420b910]
[603505.844] (EE) 10: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (dix_main+0x11a4) [0x10425c7d4]
[603505.844] (EE) 11: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (server_thread+0x1e8) [0x10404cc14]
[603505.844] (EE) 12: /usr/lib/system/libsystem_pthread.dylib (_pthread_start+0x94) [0x18599506c]
[603505.844] (EE) 13: /usr/lib/system/libsystem_pthread.dylib (thread_start+0x8) [0x18598fe2c]
... multiple places re-using the scratch space, corrupting what COMPOSITE might want from it ...
[603505.880] FreePixmap: pPixmap = 0x124039560
[603505.880] (EE)
[603505.880] (EE) Backtrace:
[603505.880] (EE) 0: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (xorg_backtrace+0x190) [0x10479ab6c]
[603505.880] (EE) 1: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (FreePixmap+0x4c) [0x1042fbe94]
[603505.880] (EE) 2: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (fbDestroyPixmap+0xa8) [0x10410dda4]
[603505.880] (EE) 3: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (damageDestroyPixmap+0x284) [0x104404998]
[603505.880] (EE) 4: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (ShmDestroyPixmap+0x1c0) [0x1044cadf8]
[603505.880] (EE) 5: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (compFreeOldPixmap+0x1a8) [0x1047fd0dc]
[603505.881] (EE) 6: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (compResizeWindow+0x220) [0x1047fd368]
[603505.881] (EE) 7: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (ConfigureWindow+0x33d0) [0x10435a1dc]
[603505.881] (EE) 8: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (ProcConfigureWindow+0x40c) [0x104214500]
[603505.881] (EE) 9: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (Dispatch+0x8bc) [0x10420b910]
[603505.881] (EE) 10: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (dix_main+0x11a4) [0x10425c7d4]
[603505.881] (EE) 11: /Applications/Utilities/XQuartz.app/Contents/MacOS/X11.bin (server_thread+0x1e8) [0x10404cc14]
[603505.881] (EE) 12: /usr/lib/system/libsystem_pthread.dylib (_pthread_start+0x94) [0x18599506c]
[603505.881] (EE) 13: /usr/lib/system/libsystem_pthread.dylib (thread_start+0x8) [0x18598fe2c]
... and then the next use blows up as a use after free
==30493==ERROR: AddressSanitizer: heap-use-after-free on address 0x000124039562 at pc 0x00010417c4fc bp 0x00016c01d020 sp 0x00016c01d018
WRITE of size 1 at 0x000124039562 thread T4
#0 0x10417c4f8 in miModifyPixmapHeader miscrinit.c:74
#1 0x1042fb75c in GetScratchPixmapHeader pixmap.c:70
#2 0x104057b78 in RootlessStartDrawing rootlessCommon.c:165
#3 0x104156f98 in miPaintWindow miexpose.c:451
#4 0x104159964 in miWindowExposures miexpose.c:388
#5 0x1041a9c34 in miHandleValidateExposures miwindow.c:224
#6 0x104342678 in MapWindow window.c:2689
#7 0x104213634 in ProcMapWindow dispatch.c:913
#8 0x10420b90c in Dispatch dispatch.c:551
#9 0x10425c7d0 in dix_main main.c:272
#10 0x10404cc10 in server_thread quartzStartup.c:65
#11 0x185995068 in _pthread_start+0x90 (libsystem_pthread.dylib:arm64e+0x7068)
#12 0x18598fe28 in thread_start+0x4 (libsystem_pthread.dylib:arm64e+0x1e28)
0x000124039562 is located 2 bytes inside of 104-byte region [0x000124039560,0x0001240395c8)
freed by thread T4 here:
#0 0x10540ade4 in wrap_free+0x98 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x3ede4)
#1 0x1042fbed8 in FreePixmap pixmap.c:144
#2 0x10410dda0 in fbDestroyPixmap fbpixmap.c:94
#3 0x104404994 in damageDestroyPixmap damage.c:1504
#4 0x1044cadf4 in ShmDestroyPixmap shm.c:260
#5 0x1047fd0d8 in compFreeOldPixmap compwindow.c:385
#6 0x1047fd364 in compResizeWindow compwindow.c:418
#7 0x10435a1d8 in ConfigureWindow window.c:2411
#8 0x1042144fc in ProcConfigureWindow dispatch.c:984
#9 0x10420b90c in Dispatch dispatch.c:551
#10 0x10425c7d0 in dix_main main.c:272
#11 0x10404cc10 in server_thread quartzStartup.c:65
#12 0x185995068 in _pthread_start+0x90 (libsystem_pthread.dylib:arm64e+0x7068)
#13 0x18598fe28 in thread_start+0x4 (libsystem_pthread.dylib:arm64e+0x1e28)
previously allocated by thread T4 here:
#0 0x10540b074 in wrap_calloc+0x9c (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x3f074)
#1 0x1042fbdbc in AllocatePixmap pixmap.c:128
#2 0x10410d580 in fbCreatePixmap fbpixmap.c:54
#3 0x1042fb6d4 in GetScratchPixmapHeader pixmap.c:64
#4 0x104057b78 in RootlessStartDrawing rootlessCommon.c:165
#5 0x104156f98 in miPaintWindow miexpose.c:451
#6 0x104159964 in miWindowExposures miexpose.c:388
#7 0x1041a9c34 in miHandleValidateExposures miwindow.c:224
#8 0x104342678 in MapWindow window.c:2689
#9 0x104213634 in ProcMapWindow dispatch.c:913
#10 0x10420b90c in Dispatch dispatch.c:551
#11 0x10425c7d0 in dix_main main.c:272
#12 0x10404cc10 in server_thread quartzStartup.c:65
#13 0x185995068 in _pthread_start+0x90 (libsystem_pthread.dylib:arm64e+0x7068)
#14 0x18598fe28 in thread_start+0x4 (libsystem_pthread.dylib:arm64e+0x1e28)
Thread T4 created by T0 here:
#0 0x105404c5c in wrap_pthread_create+0x54 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x38c5c)
#1 0x10404c918 in create_thread quartzStartup.c:77
#2 0x10404c530 in QuartzInitServer quartzStartup.c:94
#3 0x104026fe0 in X11ApplicationMain X11Application.m:798
#4 0x104032a9c in X11ControllerMain X11Controller.m:922
#5 0x10404d008 in server_main quartzStartup.c:135
#6 0x10401b198 in do_start_x11_server bundle-main.c:396
#7 0x104016148 in _Xstart_x11_server mach_startupServer.c:192
#8 0x10401734c in mach_startup_server mach_startupServer.c:402
#9 0x18595da00 in mach_msg_server+0x1a0 (libsystem_kernel.dylib:arm64e+0x8a00)
#10 0x10401b8b4 in main bundle-main.c:713
#11 0x18566be4c (<unknown module>)
SUMMARY: AddressSanitizer: heap-use-after-free miscrinit.c:74 in miModifyPixmapHeader
Edited by Jeremy Huddleston Sequoia