Fully loadable input/output drivers
The API between DIX and DDX is pretty thin. Here's the total set of exported functions from hw/vfb/:
datura:~/git/xserver% nm -a --size build/hw/vfb/*/*.o | LC_ALL=C grep ' [A-Z] '
0000000000000001 T DDXRingBell
0000000000000005 T CloseInput
0000000000000005 T ProcessInputEvents
0000000000000006 T LegalModifier
0000000000000095 T InitInput
0000000000000001 T OsVendorFatalError
0000000000000001 T OsVendorInit
0000000000000005 T AbortDDX
0000000000000078 T ddxUseMsg
0000000000000163 T ddxGiveUp
00000000000001b5 T InitOutput
00000000000004bb T ddxProcessArgument
So, a few stubs, some hooks for logging and error handling, and some big magical "initialize all the things" entrypoints. Those are big/magical partly because dix assumes that input/output combinations are likely for specific workstations. This suggests that with some gentle rearrangement the input and output components could be usefully factored out. This may not be super useful for Xvfb, but: Xdmx and Xorg both know about evdev, Xorg and Xwayland both know a lot about gbm and EGL, Xdmx and Xnest both know very similar things about being X clients, etc.
The big picture view here is, imagine building one X server binary that can plug together nearly arbitrary input and output sources, with a straightforward explicit configuration (not xfree86's please, at least not at the top level). A decent amount of code motion and cleanup would be necessary to really make this viable, see linked issues for more details.