Design problems for DRM leasing via Xwayland

For context on DRM leasing and its design on Wayland, see this thread and the follow up patches, thru version 5 (latest).

I have a WIP patch here which mostly works for xrgears in its direct mode:

https://git.sr.ht/~sircmpwn/xserver/commit/25fc8b63a6f20a9b778e9816e55f3438ab04dba8

The problem is that I've ended up hardcoding my VR headset's desired mode into both Xwayland and xrgears. xrgears goes through X11 to enumerate display modes for setting up its swapchain, which naturally breaks if the output has no modes. Currently Xwayland uses the wl_output's width & height, but for most leasable connectors we don't expect to have a corresponding wl_output. The width & height on the wl_output are not known to be reliable, in any case.

Not only does $random-software-in-the-wild enumerate modes via X for use with DRM leasing, but so does Mesa's Vulkan WSI implementation:

https://gitlab.freedesktop.org/mesa/mesa/blob/master/src/vulkan/wsi/wsi_common_display.c#L2241-2281

This means that even if xrgears were to use VkDisplayKHR for enumerating modes, they would not receive the correct modes (or even any modes at all). We could patch Vulkan to use the leased DRM file descriptor to enumerate modes once the lease is issued... except we cannot. The Vulkan API is designed in two steps: first you pass it an RROutput and you receive a VkDisplayKHR, then you pass this into VK_EXT_acquire_xlib_display. In between these calls, Vulkan does not have a DRM lease, but you can still query it for modes.

After much thought, I think the best course of action is to have Xwayland open the DRM device and scan the connectors itself. We have the connector's DRM ID available when we rig up the xwl_output, so we could correlate it with the DRM connector, enumerate its modes, and rig them up the same way the modesetting driver does, and it should Just Work. The questions then become: how do we determine which DRM card to open? We could do it the same way as the modesetting driver (default to card0 or override with an environment variable), but that's not very robust. We could extend the DRM leasing protocol to include the path to the card to be opened, but that's also not great. We also have to consider the case of multiple GPUs. The DRM object IDs are unique - should we just enumerate all of the cards and look through all of their DRM objects? If so, how do we enumerate the cards? Do we bring in udev? Suddenly we're looking at a pretty sizable change.

For now I plan on doing something for the proof-of-concept stage which copies the modesetting driver's approach for identifying the card to use, but I want to open the floor to discussion before settling on a final appropach.