DP_DPCD_QUIRK_HBLANK_EXPANSION_REQUIRES_DSC broke some hardware combination

Background

My Lenovo X1 Carbon Gen9 (i7-1185G7 with Iris Xe) recently updated from kernel 6.7.12 to 6.9.10, and my dual 4K@60 setup no longer works. The monitors are correctly enumerated, but KMS will fail and fallback to previous state (disabled by default).

I have the following setup:

X1 Carbon --> TB4 port (in DP alt mode) --> DP MST hub to 2x HDMI --> 2x HDMI --> 2x Dell S2721Q

X1 Carbon --> TB4 port --> Anker 777 Dock --> 2x HDMI --> 2x Dell S2721Q

The DP MST hub is this one: https://www.amazon.com/gp/product/B0CC2KP5W8/ , with Synaptics chip. I haven't looked at the debug dump for Anker 777 at work, but I believe it has a Synaptics MST chip as well. I can confirm tomorrow.

Investigation

I reverted my kernel to 6.7.12, and KMS worked again for dual 4K@60. --> Confirmed it is KMS issue, not other userland updates.

Using other modes, for example, dual 4K@30 worked in both kernel version.

Tried to avoid MST by using the native HDMI port and a USB-C to HDMI adapter, dual 4K@60 worked. I also confirmed that the EDID in native HDMI port reports standard CEA-861 VIC 97 timing, while plugged in the DP MST hub it reports some non-standard timing similar to CVT-RB2.

Tried the latest Debian live CD with kernel 6.10.4. The 1st output of DP MST hub still only runs 4K@30, but now the 2nd output runs 4k@60.

Tried to disable the quirk with a kernel patch:

--- linux-6.10.4.orig/drivers/gpu/drm/display/drm_dp_helper.c
+++ linux-6.10.4/drivers/gpu/drm/display/drm_dp_helper.c
@@ -2279,8 +2279,8 @@ static const struct dpcd_quirk dpcd_quir
        { OUI(0x00, 0x00, 0x00), DEVICE_ID('C', 'H', '7', '5', '1', '1'), false, BIT(DP_DPCD_QUIRK_NO_SINK_COUNT) },
        /* Synaptics DP1.4 MST hubs can support DSC without virtual DPCD */
        { OUI(0x90, 0xCC, 0x24), DEVICE_ID_ANY, true, BIT(DP_DPCD_QUIRK_DSC_WITHOUT_VIRTUAL_DPCD) },
-       /* Synaptics DP1.4 MST hubs require DSC for some modes on which it applies HBLANK expansion. */
-       { OUI(0x90, 0xCC, 0x24), DEVICE_ID_ANY, true, BIT(DP_DPCD_QUIRK_HBLANK_EXPANSION_REQUIRES_DSC) },
+//     /* Synaptics DP1.4 MST hubs require DSC for some modes on which it applies HBLANK expansion. */
+//     { OUI(0x90, 0xCC, 0x24), DEVICE_ID_ANY, true, BIT(DP_DPCD_QUIRK_HBLANK_EXPANSION_REQUIRES_DSC) },
        /* MediaTek panels (at least in U3224KBA) require DSC for modes with a short HBLANK on UHBR links. */
        { OUI(0x00, 0x0C, 0xE7), DEVICE_ID_ANY, false, BIT(DP_DPCD_QUIRK_HBLANK_EXPANSION_REQUIRES_DSC) },
        /* Apple MacBookPro 2017 15 inch eDP Retina panel reports too low DP_MAX_LINK_RATE */

Also fixed dual 4K@60 on kernel 6.10.4. --> Confirmed it is the quirk WA causing issues.

I tried with a different laptop (Ultra 155H with Arc), doesn't exhibit the same issue. At this point I suspect it has something to do with bandwidth allocation, or DSC implementation?

I'm not familiar with DRM drivers but I am a software engineer and do some kernel programming at work. Please let me know how can I help you collect further information for debug. Thank you!