Displays behind MST hubs non-functional (regression in kernel 6.1)
Brief summary of the problem:
TBT MST stops working after s2idle and further suspend attempts freeze the system:
[ 40.504804] amdgpu 0000:63:00.0: [drm] *ERROR* No payload for [MST PORT:0000000006da5d29] found in mst state 00000000b0e556d6
Hardware description:
- CPU: AMD Ryzen 7 6800U with Radeon Graphics
- GPU: VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:1681] (rev c4)
- Type of Display Connection: HDMI over TBT MST
System information:
- Distro name and Version: Ubuntu Focal
- Kernel version: 6.0.0-060000rc7drmtip20220927-generic
How to reproduce the issue:
- Connect system to a TBT dock with HDMI display
- Suspend/resume the system
- No image on display image, and we can found error in dmesg:
[ 40.504804] amdgpu 0000:63:00.0: [drm] *ERROR* No payload for [MST PORT:0000000006da5d29] found in mst state 00000000b0e556d6 - The next system suspend fails - power button is blinking (i.e. LPI works) but the eDP is still lit up.
Attached files:
Activity
mentioned in commit lyudess/linux@8d27e84a
Mhhhhhh, not sure how exactly this is happening quite yet but so far it seems like there's a good chance either amdgpu is disabling a CRTC twice somehow, or there's some part of the atomic state we're not checking in the MST helpers to prevent us from removing the payload too early. Since earlier up in the log I see:
[ 37.289813] [drm:dm_update_crtc_state [amdgpu]] Disabling DRM crtc: 67 [ 37.290103] [drm:dm_update_crtc_state [amdgpu]] Disabling DRM crtc: 70 [ 37.290388] amdgpu 0000:63:00.0: [drm:drm_dp_mst_atomic_check_payload_alloc_limits [drm_display_helper]] [MST PORT:000000008eceac25] releases all time slots [ 37.290401] amdgpu 0000:63:00.0: [drm:drm_dp_mst_atomic_check_payload_alloc_limits [drm_display_helper]] [MST MGR:000000001f11f8c8] mst state 00000000823e2989 TU pbn_div=0 avail=63 used=0 [ 37.290418] amdgpu 0000:63:00.0: [drm:drm_atomic_commit [drm]] committing 0000000095014e92Followed later down by (right below the WARN):
[ 45.165735] amdgpu 0000:63:00.0: [drm] *ERROR* No payload for [MST PORT:000000008eceac25] found in mst state 00000000a93ed26b [ 45.165737] amdgpu 0000:63:00.0: [drm:drm_atomic_helper_check_modeset [drm_kms_helper]] [CONNECTOR:129:DP-8] driver check failed [ 45.165745] [drm:amdgpu_dm_atomic_check [amdgpu]] drm_atomic_helper_check_modeset() failed [ 45.165912] [drm:amdgpu_dm_atomic_check [amdgpu]] Atomic check failed with err: -22Mid-way through typing this comment I was looking at amdgpu's driver code, and immediately noticed a problem I apparently introduced by mistake
Thanks. I think the issue is resolved but I am not sure, because amdgpu now hits another bug. Here's the splat:dmesg.txt
mentioned in issue #2064 (closed)
I suspect that this issue is closely related to #1926 (closed) . I've seen the guilty error messages for both issues pop up while experimenting with my new ThinkPad T14 Gen 2 AMD, running EndeavourOS with a KDE Plasma Wayland session.
The issue seems to stem from the exact combination of AMD integrated graphics, USB-C display output, KDE, Wayland and a high resolution/high refresh rate monitor:
- The display works directly over HDMI.
- The display works on GNOME with Wayland.
- The display works on KDE with X11.
- Displays seem to work if they are purposely downscaled, i.e. to 1080p from 4K.
I have access to a wide variety of potentially affected monitors and noticed a wide variety of results:
- A Dell 4K USB-C monitor works about half the time, doesn't work the other half, and doesn't work coming out of sleep mode.
- A Samsung 4K USB-C monitor causes the system to become unresponsive and the laptop display to start flickering until unplugged. I notice that the display rapidly connects and disconnects in the display settings.
- An LG UltraGear 1440p 144hz monitor connected over a ThinkPad Thunderbolt 3 dock doesn't work at all. It works over the HDMI port (but is capped at 100hz due to monitor limitations over HDMI.)
- A Sceptre ultra-wide 1440p 144hz monitor connected over a ThinkPad Ultra Dock works.
- Two typical 1080p monitors connected over a ThinkPad Thunderbolt 3 dock (with MST) work.
Let me know if there's any logs I can provide to help or if there's someone I can chuck a beer at to get this looked at.
Edited by Swift LokeThe Thunderbolt 3 dock is an MST hub— I've previously had to mess around with the DP settings on that LG monitor, but setting it to either 1.2 or 1.4 yields the same result. Two of the monitors listed connect directly over USB-C, and the last one worked fine.
Every scenario can successfully connect with either a different computer or one of the workaround scenarios I describe above.
Edited by Swift Loke
-
So - after trying and failing to reproduce this locally, it occurred to me that we all may have made a mistake here. I realized there was a patch I received and applied from Intel before:
5d832b6694e0 ("drm/dp_mst: Avoid deleting payloads for connectors staying enabled")
It's possible this isn't the actual cause of the problem but I think it's worth double checking y'all are testing a branch with this commit added if y'all wouldn't mind, it should be in drm-tip.
I'm happy to try out anything you need. Unfortunately, I missed what you are saying, as I'm not incredibly familiar with kernel or GPU driver development— what are you asking me to do here? Are you asking me to build amdgpu with a certain patch applied that isn't in mainline yet? Or are you asking me to build something without this patch applied, as it was mainlined?
Further direction would be much appreciated... Sorry for not knowing what I'm doing ;)
(If there's an FAQ, IRC group or mailing list I can hop on to receive any support necessary to accomplish this, I can go there instead— I'm positive that every one of you all are sick of answering these kinds of questions.)
Edited by Swift Loke-
Thanks for letting me know - unfortunately I did some searching today and while I have many amd machines apparently not a single one is thunderbolt capable :(.
I think I've got enough info to go off from this, but it'd be helpful if you could run the dmesg you got through
./scripts/decode_stacktrace.shin the kernel source tree so it can decode the line numbers for the symbols in the splat. I have a bad feeling though this is going to end up being something to do with amdgpu not passing -EDEADLK down through functions correctly as a result of using bool function signatures all over the place, but we'll see -
yeah I definitely think I had the right guess in the above comment. Could you try this patch and see if it fixes the issue?
Same kernel splat can be observed with the branch:
-
Ah, seems I missed a couple of spots that were still using booleans so we still weren't handling deadlocks. Mind giving this patch a shot?
lyudess/linux@78ede1d0Edited by Lyude Paul -
Agh! Sorry, I went to close vim and realized I forgot to save one of the files before committing
Presumably you meant lyudess/linux@5f3ae37f (1bf066b6~1)
I did try the 54f3ae37f commit on my laptop, but I still have hard locks when I try to use 2 HDMI monitors on a external USB C dock on my Thinkpad T14s.
Specs:
OS: Arch Linux x86_64 Host: 21CQ000GUS ThinkPad T14s Gen 3 Kernel: 6.1.0-rc3-1-mainline-custom Uptime: 4 hours, 39 mins Packages: 1359 (pacman), 10 (flatpak) Shell: zsh 5.9 Resolution: 1920x1080 DE: GNOME 43.0 WM: Mutter WM Theme: Adwaita Theme: adw-gtk3-dark [GTK2/3] Icons: Adwaita [GTK2/3] Terminal: gnome-terminal CPU: AMD Ryzen 7 PRO 6850U with Radeon Graphics (16) @ 4.768GHz GPU: AMD ATI Radeon 680M Memory: 16040MiB / 30851MiBKernel log:
Oct 31 18:56:26 thinkryzen kernel: [drm] DP Alt mode state on HPD: 1 Oct 31 18:56:26 thinkryzen kernel: [drm] DM_MST: starting TM on aconnector: 00000000c7dc8ca9 [id: 95] Oct 31 18:56:26 thinkryzen boltd[555]: probing: started [1000] Oct 31 18:56:26 thinkryzen kernel: [drm] Downstream port present 1, type 2 Oct 31 18:56:26 thinkryzen kernel: [drm] Downstream port present 1, type 2 Oct 31 18:56:27 thinkryzen kernel: ------------[ cut here ]------------ Oct 31 18:55:56 thinkryzen kernel: audit: type=1131 audit(1667260556.724:185): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=geoclue comm="systemd" ex> Oct 31 18:55:58 thinkryzen gnome-shell[1494]: meta_display_get_monitor_geometry: assertion 'monitor >= 0 && monitor < n_logical_monitors' failed Oct 31 18:55:58 thinkryzen gnome-shell[1494]: meta_display_get_monitor_geometry: assertion 'monitor >= 0 && monitor < n_logical_monitors' failed Oct 31 18:55:58 thinkryzen gnome-shell[1494]: meta_display_get_monitor_geometry: assertion 'monitor >= 0 && monitor < n_logical_monitors' failed Oct 31 18:55:58 thinkryzen gnome-shell[1494]: meta_display_get_monitor_geometry: assertion 'monitor >= 0 && monitor < n_logical_monitors' failed Oct 31 18:56:00 thinkryzen kernel: [drm] DP Alt mode state on HPD: 1 Oct 31 18:56:01 thinkryzen kernel: Registered IR keymap rc-cec Oct 31 18:56:01 thinkryzen kernel: rc rc0: DP-2 as /devices/pci0000:00/0000:00:08.1/0000:33:00.0/rc/rc0 Oct 31 18:56:01 thinkryzen kernel: input: DP-2 as /devices/pci0000:00/0000:00:08.1/0000:33:00.0/rc/rc0/input36 Oct 31 18:56:01 thinkryzen boltd[555]: probing: started [1000] Oct 31 18:56:01 thinkryzen systemd-logind[484]: Watching system buttons on /dev/input/event10 (DP-2) Oct 31 18:56:03 thinkryzen dbus-daemon[1433]: [session uid=1000 pid=1433] Activating via systemd: service name='org.gtk.vfs.Metadata' unit='gvfs-metadata.ser> Oct 31 18:56:03 thinkryzen kernel: usb 9-1.3: new full-speed USB device number 10 using xhci_hcd Oct 31 18:56:03 thinkryzen systemd[1409]: Starting Virtual filesystem metadata service... Oct 31 18:56:03 thinkryzen dbus-daemon[1433]: [session uid=1000 pid=1433] Successfully activated service 'org.gtk.vfs.Metadata' Oct 31 18:56:03 thinkryzen systemd[1409]: Started Virtual filesystem metadata service. Oct 31 18:56:03 thinkryzen kernel: usb 9-1.3: config 1 has an invalid interface number: 3 but max is 2 Oct 31 18:56:03 thinkryzen kernel: usb 9-1.3: config 1 has an invalid interface number: 3 but max is 2 Oct 31 18:56:03 thinkryzen kernel: usb 9-1.3: config 1 has an invalid interface number: 3 but max is 2 Oct 31 18:56:03 thinkryzen kernel: usb 9-1.3: config 1 has no interface number 2 Oct 31 18:56:03 thinkryzen kernel: usb 9-1.3: New USB device found, idVendor=262a, idProduct=9023, bcdDevice= 0.01 Oct 31 18:56:03 thinkryzen kernel: usb 9-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=0 Oct 31 18:56:03 thinkryzen kernel: usb 9-1.3: Product: DAC Audio Oct 31 18:56:03 thinkryzen kernel: usb 9-1.3: Manufacturer: E+ Corp. Oct 31 18:56:03 thinkryzen kernel: hid-generic 0003:262A:9023.0007: No inputs registered, leaving Oct 31 18:56:03 thinkryzen kernel: hid-generic 0003:262A:9023.0007: hidraw4: USB HID v1.00 Device [E+ Corp. DAC Audio] on usb-0000:34:00.4-1.3/input0 Oct 31 18:56:04 thinkryzen mtp-probe[2466]: checking bus 9, device 10: "/sys/devices/pci0000:00/0000:00:08.3/0000:34:00.4/usb9/9-1/9-1.3" Oct 31 18:56:04 thinkryzen mtp-probe[2466]: bus: 9, device: 10 was not an MTP device Oct 31 18:56:04 thinkryzen mtp-probe[2473]: checking bus 9, device 10: "/sys/devices/pci0000:00/0000:00:08.3/0000:34:00.4/usb9/9-1/9-1.3" Oct 31 18:56:04 thinkryzen mtp-probe[2473]: bus: 9, device: 10 was not an MTP device Oct 31 18:56:04 thinkryzen systemd[1409]: Reached target Sound Card. Oct 31 18:56:04 thinkryzen dbus-daemon[483]: [system] Activating via systemd: service name='org.freedesktop.Avahi' unit='dbus-org.freedesktop.Avahi.service' > Oct 31 18:56:04 thinkryzen dbus-daemon[483]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.Avahi.service': Unit dbus-org.freedesktop.> Oct 31 18:56:04 thinkryzen rtkit-daemon[1141]: Supervising 7 threads of 3 processes of 1 users. Oct 31 18:56:04 thinkryzen rtkit-daemon[1141]: Successfully made thread 2475 of process 1632 owned by '1000' RT at priority 5. Oct 31 18:56:04 thinkryzen rtkit-daemon[1141]: Supervising 8 threads of 3 processes of 1 users. Oct 31 18:56:06 thinkryzen boltd[555]: probing: timeout, done: [2095147] (2000000) Oct 31 18:56:24 thinkryzen gnome-shell[1494]: meta_display_get_monitor_geometry: assertion 'monitor >= 0 && monitor < n_logical_monitors' failed Oct 31 18:56:24 thinkryzen gnome-shell[1494]: meta_display_get_monitor_geometry: assertion 'monitor >= 0 && monitor < n_logical_monitors' failed Oct 31 18:56:24 thinkryzen gnome-shell[1494]: meta_display_get_monitor_geometry: assertion 'monitor >= 0 && monitor < n_logical_monitors' failed Oct 31 18:56:24 thinkryzen gnome-shell[1494]: meta_display_get_monitor_geometry: assertion 'monitor >= 0 && monitor < n_logical_monitors' failed Oct 31 18:56:26 thinkryzen kernel: [drm] DP Alt mode state on HPD: 1 Oct 31 18:56:26 thinkryzen kernel: [drm] DM_MST: starting TM on aconnector: 00000000c7dc8ca9 [id: 95] Oct 31 18:56:26 thinkryzen boltd[555]: probing: started [1000] Oct 31 18:56:26 thinkryzen kernel: [drm] Downstream port present 1, type 2 Oct 31 18:56:26 thinkryzen kernel: [drm] Downstream port present 1, type 2 Oct 31 18:56:27 thinkryzen kernel: ------------[ cut here ]------------ Oct 31 18:56:27 thinkryzen kernel: WARNING: CPU: 0 PID: 1494 at drivers/gpu/drm/drm_modeset_lock.c:317 drm_modeset_lock+0xcd/0xe0 Oct 31 18:56:27 thinkryzen kernel: Modules linked in: rfcomm ccm michael_mic xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink > Oct 31 18:56:27 thinkryzen kernel: crc32_pclmul amdgpu uvcvideo snd_rpl_pci_acp6x polyval_clmulni snd_intel_dspcfg videobuf2_vmalloc mac80211 snd_intel_sdw_> Oct 31 18:56:27 thinkryzen kernel: i8042 nvme_common xhci_pci_renesas serio Oct 31 18:56:27 thinkryzen kernel: CPU: 0 PID: 1494 Comm: gnome-shell Not tainted 6.1.0-rc3-1-mainline-custom #1 a2aaa7fb81f8dd1feee143f3d88c83d5287c37bf Oct 31 18:56:27 thinkryzen kernel: Hardware name: LENOVO 21CQ000GUS/21CQ000GUS, BIOS R22ET55W (1.25 ) 09/14/2022 Oct 31 18:56:27 thinkryzen kernel: RIP: 0010:drm_modeset_lock+0xcd/0xe0 Oct 31 18:56:27 thinkryzen kernel: Code: ff ff ff eb d5 e8 e3 5f 49 00 eb 91 0f 0b e9 75 ff ff ff 83 f8 8e 74 c0 83 f8 dd 75 bd 48 89 6b 18 c7 43 20 00 00 00> Oct 31 18:56:27 thinkryzen kernel: RSP: 0018:ffffaab506697728 EFLAGS: 00010286 Oct 31 18:56:27 thinkryzen kernel: RAX: 0000000000000000 RBX: ffffaab506697c80 RCX: 0000000000000000 Oct 31 18:56:27 thinkryzen kernel: RDX: ffff952053c18000 RSI: ffffaab506697c80 RDI: ffff952055b46550 Oct 31 18:56:27 thinkryzen kernel: RBP: ffff952055b46550 R08: ffff952055b46540 R09: ffffaab506697a14 Oct 31 18:56:27 thinkryzen kernel: R10: ffff9520aa541a00 R11: ffff952055afe800 R12: ffff952055b46578 Oct 31 18:56:27 thinkryzen kernel: R13: ffff95200cd4e400 R14: ffff9520ca280000 R15: ffff952055b46000 Oct 31 18:56:27 thinkryzen kernel: FS: 00007f33767dab00(0000) GS:ffff95271ee00000(0000) knlGS:0000000000000000 Oct 31 18:56:27 thinkryzen kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Oct 31 18:56:27 thinkryzen kernel: CR2: 00007fd79c016008 CR3: 0000000153c08000 CR4: 0000000000750ef0 Oct 31 18:56:27 thinkryzen kernel: PKRU: 55555554 Oct 31 18:56:27 thinkryzen kernel: Call Trace: Oct 31 18:56:27 thinkryzen kernel: <TASK> Oct 31 18:56:27 thinkryzen kernel: drm_atomic_get_private_obj_state+0x5c/0x160 Oct 31 18:56:27 thinkryzen kernel: compute_mst_dsc_configs_for_link+0x59/0x9e0 [amdgpu eb77160797c5f4cb03fc88190c3f041934cb0319]I'm willing to test if there are other commits I can try. The problem is reproducer 100%. If I use a single HDMI connection I don't have crashes, but sometimes the image on the monitor goes dark for a few seconds. I did try Windows for a little bit and it seemed to work, so I'm guessing this is not a hardware problem.
Edited by Nicolas Frenay-
Could you post the entire kernel log, preferably after booting with
drm.debug=0x16 log_buf_len=50Madded to the kernel commandline? (fwiw, you can just use dmesg orjournalctl --dmesg). It's hard to tell if there's still some spot I'm missing in the patch from this backtrace, since it seems like we're still hitting the same issue of the lock list somehow not being empty when we expect it to be.If you're willing to as well and know how: it'd definitely be really useful to have someone test this with KASAN (inline instrumentation specifically) turned on. Since unfortunately it's still not clear to me if
list_empty()here is actually hitting a populated list or if it's hitting uninitialized memory. Here's the full kernel log by journalctl, but without setting the drm.debug and log_buf_len. I'll add those later today and do another test and report back here with new logs.
About KASAN, I've never tried that before but I should be able to do it. Any tips on using it? I've quickly read I have to rebuild the kernel with CONFIG_KASAN = y, but not sure if there are any specifics to whitelist some of the files on the stacktrace.
Log with drm.debug=0x16 log_buf_len=50M.
Additional info on reproducer for this log:
- Booted with eDP (internal LCD) + 1 external HDMI plugged on the USB C hub
- Plugged 2nd HDMI monitor on the same USB C hub -> crash
Tried also booting with both HDMI attached but it crashes when the graphical UI would be shown. Caps-lock is still responding, but I can't go to a terminal (Ctrl + Alt + Fn).
I'll start rebuilding the kernel with KASAN.
Ok, here's the log with
drm.debug=0x16 log_buf_len=50Mand KASAN.Hope it helps debugging. Let me know if I can help with anything else. crash-20221101-full-debug-kasan.txt
Edited by Nicolas Frenay-
Something very screwy is happening here :, this doesn't seem to be a deadlock like I thought it might be. Doing a bit of digging still, what's really confusing to me about this issue is that it almost looks like that the lock we're trying to acquire is somehow both already in the lock acquisition list, but it seems like
ww_mutex()still returns 0 when we grab the actual ww lock despite the fact if it's in the acquisition list we should get-EAGAIN.@khfeng @nicolas.frenay if you guys could get a dmesg with lockdep turned on as wel, including the bits for wounded warrior mutexes, that would be useful in the mean time as maybe lockdep will tell us something new about this. Or if you have any guesses on where we could go wrong grabbing the modesetting lock for a private object here, that'd help too!
-
Also, pushed another patch to my branch. I don't know if this is going to fix things as it's still not even really clear to me what's happening here. Please give another try with these two patches applied (on top of one another):
lyudess/linux@cfb7d72b lyudess/linux@c4a3bc1a
And please make sure to get the lockdep enabled log like I mentioned when you have the chance
Ok, I'm rebuilding the kernel with lockdeps options and the c4a3bc1a patch as well.
About the params, maybe I'm doing some silly mistake, but I did set those. Do I need aditional kernel configs to be able to use them?
See:
cat /proc/cmdline BOOT_IMAGE=/vmlinuz-linux-mainline-custom-kasan root=UUID=3653ffb5-4905-4df8-b747-0e49ebb41ba9 rw drm.debug=0x16 log_buf_len=50MUnfortunatelly not, it was the right log.
Here's the latest log with the last patch, lockdeps (I hope I flagged the right configs) and kernel params. This time I didn't filter only dmesg, in case that helps.
mentioned in issue #2210 (closed)
marked #2210 (closed) as a duplicate of this issue
marked this issue as related to #2210 (closed)
mentioned in issue #2163
To try to repro this problem I have a Z13 with a Lenovo TBT3 dock (with MST hub) and DP-DVI cable connected to a DVI monitor.
- Plain 6.1-rc3 got me the splat and NULL pointer error as mentioned above.
- So I built a 6.1-rc3 with lyudess/linux@cfb7d72b and lyudess/linux@c4a3bc1a applied.
For me it's working now. Here's the log from the moment the DP cable is hotplugged into the dock (kernel has lockdep set, compiled with KASAN, and kernel command line has
drm.debug=0x16). dmesg_hotplug.txt-
Could you double check with the commits on top of drm-tip or a more recent rc? Might be worth it seeing as it sounds like it fixed stuff on @superm1 's issue
Maybe I was just lucky because this is pretty straightforward topology with one external monitor. I'll try some more and see if I can trigger anything.
But it also seems that @nicolas.frenay still has some problems even with 6.1-rc3 with those two commits. @lyudess to me all of this stuff is looking independent of the USB4 fabric. Actually @nicolas.frenay is reproducing it using DP-alt mode according to above logs.
Can you have a try with some of the docks w/ MST you have on hand with even with older hardware? Anything Titan Ridge or newer should fall back to DP-alt mode. It seems like we're probably at "hardware independent amdgpu issue with dp mst helpers" at this point.
-
Oh yeah it's definitely helper related at this point! The USB-C statement was just a guess at the start, now though I'm more convinced it's specifically DSC related (or rather, it only triggers on setups capable of DSC regardless of if it's being used or not is my guess). Basically the problem is the only devices I have that can do DSC are USB-C only as I actually did give quite a number of tries at reproducing this already with no luck :(. There is a machine in RH's lab that may be able to reproduce this that I can try getting access to, we'll have to see.
@superm1 I only have problems when I plug a second HDMI monitor on the USB-C dock. Plugging just one monitor on either port of the USB-C dock works ok. I do have some image flickering but I don't think it's related to this issue.
Is there any other test or logs you would like me to provide? I might be able to try a different dock, and if it works I'll have to return this problematic one, which would be bad as I wouldn't be able to reproduce it anymore. I'm unfortunatelly unfamiliar with the DRM code.
Thanks for all the support.
-
Also FWIW I'll make a point to ask my OEM contact about the machine I mentioned tomorrow, I might be able to get someone to give me a ride up to the office so we can get this done sooner then later. The other option is if I can just find a DSC capable monitor with a DP hookup, I -think- that'd likely also hit the issue if I'm guessing correctly.
-
Sorry for the spam, just had an idea: I'll also see if I can give a shot at just hacking together a patch for amdgpu to force it to try to consider DSC which in theory hopefully should hit it. And yeah - I'll send out the patches tomorrow as well - both of these patches we kind of want regardless of if they fix the issue or not anyway.
-
BINGO got the backtrace :). Digging in now, will make sure the fixes I've got so far are on the ML at least by tomorrow and will figure out whatever else is still broken here.agh no i didn't but i'm getting there sorry for the false alarm!Edited by Lyude Paul @lyudess It's typo. It's 6.1-rc3 instead of 6.1-rc1.
Some additional info that might be relevant: I've used my computer today with the debug kernel, and used two different USB-C hubs. The initial hours using the one I've reported the bugs earlier, but only using one of the HDMI ports, so that I wouldn't crash. As this device had HDMI inconsistencies (blank screens every few minutes), I've changed to another USB-C device that only has 1 HDMI port but has a consistent image and used it for some hours.
I now had a crash while coding in Intellij, unrelated to plugging another HDMI device, and went to the kernel logs and found that during the day I have a few drm_modeset_drop_locks traces thrown.
Please see the kernel logs attached: crash-20221103-traces-and-crash.txt
The system lock seems to be unrelated to this issue, as the error log is:
Nov 03 19:08:25 thinkryzen kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0 timeout, signaled seq=306735, emitted seq=306737 Nov 03 19:08:25 thinkryzen kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process pid 0 thread pid 0I'm guessing it's this issue: #2068
TL;DR: full logs for session with
- First hours: eDP (internal display) + USB-Dock with 2 hdmi ports, using 1 HDMI port to a LCD monitor (image blanks every few mins) + External LCD using USB-C port.
- Last hours: eDP (internal display) + USB-Dock with 1 hdmi port, using 1 HDMI port to a LCD monitor (image consistent, no blanking) + External LCD using USB-port.
And here's
ddcutil detect:Invalid display I2C bus: /dev/i2c-14 DRM connector: card0-eDP-1 EDID synopsis: Mfg id: IVO - UNK Model: Product code: 35908 (0x8c44) Serial number: Binary serial number: 0 (0x00000000) Manufacture year: 2021, Week: 0 DDC communication failed This is an eDP laptop display. Laptop displays do not support DDC/CI. Display 1 I2C bus: /dev/i2c-15 DRM connector: card0-DP-1 EDID synopsis: Mfg id: DZX - UNK Model: K1301R Product code: 4880 (0x1310) Serial number: 000000000000 Binary serial number: 269488144 (0x10101010) Manufacture year: 2019, Week: 32 VCP version: 2.2 Display 2 I2C bus: /dev/i2c-16 DRM connector: card0-DP-2 EDID synopsis: Mfg id: LEN - Lenovo Group Limited Model: T24i-2L Product code: 25264 (0x62b0) Serial number: VKNB5961 Binary serial number: 16843009 (0x01010101) Manufacture year: 2022, Week: 6 VCP version: 2.2- K1301R is plugged directly via USB-C to my laptop.
- T24i-2L is plugged on the USB-C hub (either the 2 port or the 1 port)
Edited by Nicolas Frenay-
[ 285.422986] [drm] DM_MST: starting TM on aconnector: 00000000ba42b7a1 [id: 86] [ 286.442909] [drm] No DSC? Forcing [ 286.442988] ------------[ cut here ]------------ [ 286.443012] WARNING: CPU: 0 PID: 86 at drivers/gpu/drm/drm_modeset_lock.c:317 drm_modeset_lock+0x102/0x110 [drm] [ 286.443123] Modules linked in: amdgpu drm_ttm_helper ttm video iommu_v2 gpu_sched i2c_algo_bit drm_buddy drm_display_helper drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops cec rfkill vfat fat snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_codec_hdmi snd_hda_intel snd_intel_dspcfg snd_hda_codec snd_hwdep snd_hda_core ppdev intel_rapl_msr wmi_bmof snd_seq snd_seq_device snd_pcm snd_timer snd intel_rapl_common i2c_piix4 rapl soundcore k10temp wmi parport_pc parport gpio_amdpt gpio_generic acpi_cpufreq drm zram ip_tables crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ccp r8169 w83627hf_wdt fuse ipmi_devintf ipmi_msghandler [ 286.443399] CPU: 0 PID: 86 Comm: kworker/0:2 Kdump: loaded Not tainted 6.1.0-rc1Lyude-Test+ #2 [ 286.443438] Hardware name: MSI MS-7A39/A320M GAMING PRO (MS-7A39), BIOS 1.I0 01/22/2019 [ 286.443472] Workqueue: events_long drm_dp_mst_link_probe_work [drm_display_helper] [ 286.443530] RIP: 0010:drm_modeset_lock+0x102/0x110 [drm] [ 286.443621] Code: 7a dd 41 89 c4 e9 6b ff ff ff 0f 0b e9 44 ff ff ff 41 83 fc 8e 74 bb 41 83 fc dd 75 b8 48 89 6b 18 c7 43 20 00 00 00 00 eb ab <0f> 0b e9 5e ff ff ff 0f 1f 80 00 00 00 00 0f 1f 44 00 00 41 57 41 [ 286.443685] RSP: 0018:ffffbb84c0487938 EFLAGS: 00010286 [ 286.443713] RAX: 0000000000000000 RBX: ffffbb84c0487d78 RCX: ffff9cc4ecc33000 [ 286.443744] RDX: ffff9cc4c348a880 RSI: ffffbb84c0487d78 RDI: ffff9cc4ecc3c550 [ 286.443775] RBP: ffff9cc4ecc3c550 R08: ffff9cc4ecc3c850 R09: 00000000fff7ffff [ 286.443805] R10: ffffbb84c0487a20 R11: ffff9cc5de1fdfe8 R12: 0000000000000000 [ 286.443835] R13: ffff9cc4ecc3c578 R14: ffff9cc509fbe880 R15: ffff9cc4e26c1ee0 [ 286.443866] FS: 0000000000000000(0000) GS:ffff9cc5d0800000(0000) knlGS:0000000000000000 [ 286.443901] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 286.443928] CR2: 000055cf8dc052b0 CR3: 0000000118d96000 CR4: 00000000003506f0 [ 286.443974] Call Trace: [ 286.443992] <TASK> [ 286.444010] drm_atomic_get_private_obj_state+0x5c/0x160 [drm] [ 286.444090] compute_mst_dsc_configs_for_link.constprop.0+0x1a/0x90 [amdgpu] [ 286.444742] ? prb_read_valid+0x17/0x20 [ 286.444770] ? desc_read_finalized_seq+0x1f/0x70 [ 286.444796] ? _prb_read_valid+0x225/0x2a0 [ 286.444825] ? tick_nohz_tick_stopped+0x12/0x30 [ 286.444852] ? __irq_work_queue_local+0x39/0x80 [ 286.444878] ? irq_work_queue+0x39/0x50 [ 286.444901] ? __wake_up_klogd.part.0+0x56/0x80 [ 286.444926] ? vprintk_emit+0x112/0x240 [ 286.444960] ? _printk+0x48/0x4e [ 286.444984] pre_validate_dsc+0x2b1/0x340 [amdgpu] [ 286.445622] amdgpu_dm_atomic_check+0x662/0x1090 [amdgpu] [ 286.446264] ? idr_alloc_u32+0x7d/0xa0 [ 286.446290] ? __ww_mutex_lock.constprop.0+0x29/0x950 [ 286.446319] ? dm_crtc_duplicate_state+0x34/0x180 [amdgpu] [ 286.446958] ? __kmem_cache_alloc_node+0x191/0x270 [ 286.448693] ? __ww_mutex_lock.constprop.0+0x29/0x950 [ 286.450334] ? preempt_count_add+0x70/0xa0 [ 286.451909] ? _raw_spin_lock_irqsave+0x19/0x50 [ 286.453478] drm_atomic_check_only+0x5c7/0xa20 [drm] [ 286.455085] drm_atomic_commit+0x46/0xa0 [drm] [ 286.456679] ? drm_plane_get_damage_clips.cold+0x1c/0x1c [drm] [ 286.458278] drm_client_modeset_commit_atomic+0x1c8/0x200 [drm] [ 286.459865] drm_client_modeset_commit_locked+0x56/0x160 [drm] [ 286.461439] drm_client_modeset_commit+0x24/0x50 [drm] [ 286.463001] drm_fb_helper_set_par+0xa9/0xe0 [drm_kms_helper] [ 286.464516] drm_fb_helper_hotplug_event.part.0+0xae/0xd0 [drm_kms_helper] [ 286.466030] drm_kms_helper_hotplug_event+0x26/0x30 [drm_kms_helper] [ 286.467531] process_one_work+0x1e5/0x3b0 [ 286.468999] worker_thread+0x50/0x3a0 [ 286.470453] ? rescuer_thread+0x370/0x370 [ 286.471893] kthread+0xe8/0x110 [ 286.473311] ? kthread_complete_and_exit+0x20/0x20 [ 286.474725] ret_from_fork+0x22/0x30 [ 286.476137] </TASK> [ 286.477524] ---[ end trace 0000000000000000 ]--- [ 286.478906] BUG: kernel NULL pointer dereference, address: 0000000000000000 [ 286.480281] #PF: supervisor read access in kernel mode [ 286.481643] #PF: error_code(0x0000) - not-present page [ 286.482993] PGD 0 P4D 0Finally hit it! I should be able to come up with a fix from here, will submit the rest of the patches I've got as well by the end of the day regardless
FWIW, issue 2068 seems to be alleviated by adding iommu=pt in kernel command line, but it is not 100% reliable. It reduces multiple crashes per day to a few crashes per week.
The crash doesn't seem to be triggered by high GPU usage or temperature. I can play video games all day long without triggering it. However, sparse but simple OpenGL commands (like CAD) triggers it from time to time. This bug was never triggered in a browser or desktop, despite Gnome uses hardware acceleration here and there.
For issue 2210, my issue was gone after using a dual MST hub, but a triple MST hub still doesn't work.
As for 6.1 without DSC, no, it doesn't work at all. When the hub supports DSC and the driver is loaded with dcdebugmask=0x04, it constantly crashes on 6.1 and fails to output signal on 6.0.
-
YEP. Cool, we get two bug fixes here too!!! And also this is a genuinely impressive bug, because I'm literally unsure how we were managing to avoid crashing the kernel at all.
SO: while everyone's bisects, understandably, led us to 4d07b0bc403403438d9cf88450506240c5faf92f I don't think that's actually where the bug was introduced, but merely where it became noticeable. As the actual problem is any kind of code that does this:
mutex_lock(&aconnector->mst_mgr->lock);The reason this isn't correct is really subtle:
aconnectorisamdgpu_dm_connector, which is shared for all connector types including MST. However,aconnector->mst_mgris only used for the actual root MST connector in a topology - which we'll basically never really be considering for DSC. Otherwise, it's simply completely zero-initialized! However, inpre_validate_dsc()(along with any other DSC state computation functions) ->pre_compute_mst_dsc_configs_for_state()guess what we do?mutex_lock(&aconnector->mst_mgr->lock); if (!compute_mst_dsc_configs_for_link(…, &aconnector->mst_mgr, …)) { … } …Bingo! Ignoring that this code doesn't look like it should work at all now: somehow, it does. Well, ish, the moment you turn on lockdep you'll start getting warnings:
[ 66.637670] DEBUG_LOCKS_WARN_ON(lock->magic != lock)This is the wild part, I think that this code has only been working up until now because aconnector is zero-initialized. And I guess mutex_lock() manages not to explode on totally zero-initialized memory! You can actually still hit this warning before 4d07b0bc403403438d9cf88450506240c5faf92f, suggesting the bug was actually introduced way back in 8c20a1ed9b4f. Anyway, the reason this starts crashing after 4d07b0bc403403438d9cf88450506240c5faf92f is likely due to the fact there's a lot more data we're reading from the topology manager after that commit. In particular, grabbing the modesetting locks for MST and potentially adding in the atomic state.
SO - I finally now have a patch, and I am much more confident this one should fix the issue now as it seems to on my setup. Keep in mind I did all of this without an actual DSC capable setup and instead just hacked up some patches to get amdgpu to hit the broken codepaths until I could figure out what the problem is :). So, definitely need someone to confirm this fully fixes the problem before submitting:
https://gitlab.freedesktop.org/lyudess/linux/-/commits/wip/amdgpu-mst-reg-gl2171-v1
Just apply the patches from me onto your kernel and let me know if this fixes the problem. Fingers crossed!
EDIT: ALSO, probably worth mentioning that if anyone from amd has been running into inconsistent success with modesets and DSC - this patch should technically also fix a race condition, so it's very much worth trying this to see if it fixes such issues.
Edited by Lyude Paul Rebuilding my kernel with these 4 patches. Will report back in about 1hr @lyudess.
Edited by Nicolas FrenayOk, reporting back with some bad news @lyudess , or maybe we're going a little bit further. Unless I made some mistake on my build, I'm still experiencing the crash when plugging in a second HDMI monitor on the USB hub, but the log is different this time:
Nov 09 21:54:32 thinkryzen kernel: [drm] DP Alt mode state on HPD: 1 Nov 09 21:54:32 thinkryzen kernel: [drm] DM_MST: starting TM on aconnector: 000000001c1d46b1 [id: 95] Nov 09 21:54:32 thinkryzen boltd[562]: probing: started [1000] Nov 09 21:54:32 thinkryzen kernel: [drm] Downstream port present 1, type 2 Nov 09 21:54:32 thinkryzen kernel: [drm] Downstream port present 1, type 2 Nov 09 21:54:35 thinkryzen boltd[562]: probing: timeout, done: [2971957] (2000000)while testing I had
dmesg -wrunning and I was able to read until the "starting TM on aconnector" line on my screen, so the crash might have happened on the boltd probing started line above.I'm gonna do some additional tests here to see if I can get more info. It would be nice to have someone else test those patches, in case I'm having a different issue now.
Just to make sure we're on the same page, I'm running linux 6.1 RC4 with these patches applied:
- e09062c9
- a68a93ef
- e32885cd
- 37bac505
Edited by Nicolas Frenay-
Just an FYI this is actually good output to see :), if your kernel isn't segfaulting then we've definitely got the bug fixed, or at the very least are on the right track and there's multiple bugs here (considering how many patches I've racked up now, that doesn't seem too unlikely).
My one concern is we could be seeing something in DRM hanging here. I'm not sure that would actually make a difference to thunderbolt though? (unfortunately I'm not totally sure how DRM/thunderbolt work together, most of the time it's all just DP from the DRM driver's perspective - perhaps with extra bw control knobs)
@superm1 @lyudess I can SSH to the machine after the image freeze and have some additional info.
When I plug the HDMI connector, the log is as I've mentioned before:
[ 919.249174] [drm] DP Alt mode state on HPD: 1 [ 919.334194] [drm] DM_MST: starting TM on aconnector: 000000003ff8c623 [id: 94] [ 919.502545] [drm] Downstream port present 1, type 2 [ 919.577925] [drm] Downstream port present 1, type 2The frozen LCD screen shows up to the DM_MST line above.
But this time I waited longer on SSH to see if anything else happened:
[ 1106.427759] INFO: task gnome-shell:1669 blocked for more than 122 seconds. [ 1106.427786] Tainted: G W OE 6.1.0-rc4-1-mainline-nicolas-2171-v1 #2 [ 1106.427797] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1106.427805] task:gnome-shell state:D stack:0 pid:1669 ppid:1580 flags:0x00004002 [ 1106.427823] Call Trace: [ 1106.427830] <TASK> [ 1106.427840] __schedule+0xd82/0x4f70 [ 1106.427865] ? rcu_read_lock_sched_held+0x14/0x80 [ 1106.427883] ? rcu_read_lock_sched_held+0x14/0x80 [ 1106.427891] ? lock_acquire+0x457/0x520 [ 1106.427906] ? lock_release+0x5be/0xd60 [ 1106.427913] ? lockdep_hardirqs_on_prepare+0x400/0x400 [ 1106.427927] ? io_schedule_timeout+0x160/0x160 [ 1106.427940] ? lock_unpin_lock+0x3d0/0x3d0 [ 1106.427951] schedule+0x134/0x230 [ 1106.427963] schedule_preempt_disabled+0x15/0x30 [ 1106.427974] __mutex_lock+0x884/0x1200 [ 1106.427989] ? drm_dp_mst_atomic_check+0x500/0xb80 [drm_display_helper d888144e430415d7160ce1595e9a230301619d64] [ 1106.428038] ? mutex_lock_io_nested+0x10e0/0x10e0 [ 1106.428053] ? dc_dsc_compute_bandwidth_range+0x3b5/0x490 [amdgpu b8a691cf9793742358bd97b18b89bb2c8e90763d] [ 1106.428975] ? drm_dp_mst_atomic_check+0x500/0xb80 [drm_display_helper d888144e430415d7160ce1595e9a230301619d64] [ 1106.429017] drm_dp_mst_atomic_check+0x500/0xb80 [drm_display_helper d888144e430415d7160ce1595e9a230301619d64] [ 1106.429059] ? drm_dp_atomic_find_time_slots+0x28/0xb90 [drm_display_helper d888144e430415d7160ce1595e9a230301619d64] [ 1106.429102] compute_mst_dsc_configs_for_link+0x980/0x2720 [amdgpu b8a691cf9793742358bd97b18b89bb2c8e90763d] [ 1106.429990] ? dm_dp_add_mst_connector+0x490/0x490 [amdgpu b8a691cf9793742358bd97b18b89bb2c8e90763d] [ 1106.430913] ? get_page_from_freelist+0x28ef/0x34f0 [ 1106.430930] ? setup_dsc_config+0x1350/0x1350 [amdgpu b8a691cf9793742358bd97b18b89bb2c8e90763d] [ 1106.431841] ? lock_acquire+0x457/0x520 [ 1106.431859] ? rcu_read_lock_sched_held+0x14/0x80 [ 1106.431868] ? lock_acquire+0x457/0x520 [ 1106.431904] ? create_validate_stream_for_sink+0x20a/0x410 [amdgpu b8a691cf9793742358bd97b18b89bb2c8e90763d] [ 1106.432747] pre_validate_dsc+0xbed/0x12a0 [amdgpu b8a691cf9793742358bd97b18b89bb2c8e90763d] [ 1106.433235] ? compute_mst_dsc_configs_for_state+0x890/0x890 [amdgpu b8a691cf9793742358bd97b18b89bb2c8e90763d] [ 1106.433712] ? dm_update_plane_state+0xdf2/0x1fe0 [amdgpu b8a691cf9793742358bd97b18b89bb2c8e90763d] [ 1106.434426] amdgpu_dm_atomic_check+0x119e/0x3950 [amdgpu b8a691cf9793742358bd97b18b89bb2c8e90763d] [ 1106.435315] ? stack_trace_save+0x95/0xd0 [ 1106.435325] ? filter_irq_stacks+0xa0/0xa0 [ 1106.435335] ? dm_update_crtc_state+0x1cf0/0x1cf0 [amdgpu b8a691cf9793742358bd97b18b89bb2c8e90763d] [ 1106.436185] ? rcu_read_lock_bh_held+0x41/0x70 [ 1106.436204] ? dm_plane_format_mod_supported+0x42/0x320 [amdgpu b8a691cf9793742358bd97b18b89bb2c8e90763d] [ 1106.437073] ? drm_plane_check_pixel_format+0x126/0x260 [ 1106.437084] ? dm_update_crtc_state+0x1cf0/0x1cf0 [amdgpu b8a691cf9793742358bd97b18b89bb2c8e90763d] [ 1106.437992] drm_atomic_check_only+0x11b3/0x3570 [ 1106.438008] ? lock_downgrade+0x720/0x720 [ 1106.438019] drm_atomic_commit+0x100/0x2c0 [ 1106.438030] ? drm_atomic_print_new_state+0x6e0/0x6e0 [ 1106.438039] ? drm_plane_get_damage_clips.cold+0x3f/0x3f [ 1106.438054] ? drm_event_reserve_init+0x1c3/0x240 [ 1106.438069] drm_mode_atomic_ioctl+0x1457/0x2040 [ 1106.438090] ? drm_atomic_set_property+0x2b60/0x2b60 [ 1106.438102] ? bit_wait_io_timeout+0x170/0x170 [ 1106.438114] ? rcu_read_lock_sched_held+0x14/0x80 [ 1106.438132] ? rcu_read_lock_sched_held+0x14/0x80 [ 1106.438142] ? do_raw_spin_unlock+0x58/0x1f0 [ 1106.438152] ? drm_atomic_set_property+0x2b60/0x2b60 [ 1106.438164] drm_ioctl_kernel+0x1fc/0x3d0 [ 1106.438173] ? __check_object_size+0x5e/0x540 [ 1106.438185] ? drm_setversion+0x7f0/0x7f0 [ 1106.438195] ? __check_object_size+0x5e/0x540 [ 1106.438207] ? drm_atomic_set_property+0x2b60/0x2b60 [ 1106.438218] drm_ioctl+0x334/0x920 [ 1106.438230] ? drm_ioctl_kernel+0x3d0/0x3d0 [ 1106.438238] ? lock_release+0x5be/0xd60 [ 1106.438245] ? rcu_read_lock_sched_held+0x14/0x80 [ 1106.438261] ? _raw_spin_unlock_irqrestore+0x5a/0x70 [ 1106.438271] ? trace_hardirqs_on+0x31/0x110 [ 1106.438290] amdgpu_drm_ioctl+0xd2/0x1b0 [amdgpu b8a691cf9793742358bd97b18b89bb2c8e90763d] [ 1106.439051] __x64_sys_ioctl+0x131/0x1a0 [ 1106.439067] do_syscall_64+0x5f/0x90 [ 1106.439079] ? do_syscall_64+0x6b/0x90 [ 1106.439090] ? do_syscall_64+0x6b/0x90 [ 1106.439099] ? trace_hardirqs_on_prepare+0x7c/0x160 [ 1106.439110] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 1106.439122] RIP: 0033:0x7fc62df15c0f [ 1106.439131] RSP: 002b:00007fffdad5e740 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 1106.439144] RAX: ffffffffffffffda RBX: 000055bd400427f0 RCX: 00007fc62df15c0f [ 1106.439152] RDX: 00007fffdad5e7e0 RSI: 00000000c03864bc RDI: 0000000000000009 [ 1106.439160] RBP: 00007fffdad5e7e0 R08: 0000000000000015 R09: 0000000000000015 [ 1106.439167] R10: 000055bd3e102010 R11: 0000000000000246 R12: 00000000c03864bc [ 1106.439173] R13: 0000000000000009 R14: 000055bd3fd67460 R15: 000055bd43942500 [ 1106.439188] </TASK> [ 1106.439271] INFO: task fwupd:2275 blocked for more than 122 seconds. [ 1106.439280] Tainted: G W OE 6.1.0-rc4-1-mainline-nicolas-2171-v1 #2 [ 1106.439287] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1106.439293] task:fwupd state:D stack:0 pid:2275 ppid:1 flags:0x00000002 [ 1106.439307] Call Trace: [ 1106.439312] <TASK> [ 1106.439321] __schedule+0xd82/0x4f70 [ 1106.439336] ? rcu_read_lock_sched_held+0x14/0x80 [ 1106.439345] ? lock_acquire+0x457/0x520 [ 1106.439357] ? lock_release+0x5be/0xd60 [ 1106.439364] ? lockdep_hardirqs_on_prepare+0x400/0x400 [ 1106.439377] ? io_schedule_timeout+0x160/0x160 [ 1106.439390] ? lock_unpin_lock+0x3d0/0x3d0 [ 1106.439401] schedule+0x134/0x230 [ 1106.439413] schedule_preempt_disabled+0x15/0x30 [ 1106.439424] __mutex_lock+0x884/0x1200 [ 1106.439438] ? drm_dp_mst_topology_get_mstb_validated+0x26/0x140 [drm_display_helper d888144e430415d7160ce1595e9a230301619d64] [ 1106.439481] ? mutex_lock_io_nested+0x10e0/0x10e0 [ 1106.439492] ? kasan_set_track+0x25/0x30 [ 1106.439503] ? kasan_save_free_info+0x2e/0x50 [ 1106.439515] ? ____kasan_slab_free+0x169/0x1c0 [ 1106.439526] ? kmem_cache_free+0x208/0x470 [ 1106.439535] ? do_sys_openat2+0x128/0x420 [ 1106.439546] ? __x64_sys_openat+0x139/0x1e0 [ 1106.439560] ? rcu_read_lock_sched_held+0x14/0x80 [ 1106.439568] ? lock_acquire+0x457/0x520 [ 1106.439579] ? rcu_read_lock_sched_held+0x14/0x80 [ 1106.439590] ? drm_dp_mst_topology_get_mstb_validated+0x26/0x140 [drm_display_helper d888144e430415d7160ce1595e9a230301619d64] [ 1106.439627] drm_dp_mst_topology_get_mstb_validated+0x26/0x140 [drm_display_helper d888144e430415d7160ce1595e9a230301619d64] [ 1106.439668] drm_dp_mst_dpcd_read+0x7f/0x3d0 [drm_display_helper d888144e430415d7160ce1595e9a230301619d64] [ 1106.439710] drm_dp_dpcd_read+0xac/0x560 [drm_display_helper d888144e430415d7160ce1595e9a230301619d64] [ 1106.439749] ? dput+0x34/0xb70 [ 1106.439758] ? dput+0x113/0xb70 [ 1106.439768] ? drm_edp_backlight_set_level+0x120/0x120 [drm_display_helper d888144e430415d7160ce1595e9a230301619d64] [ 1106.439808] ? __fsnotify_parent+0x298/0xca0 [ 1106.439821] auxdev_read_iter+0x219/0x4e0 [drm_display_helper d888144e430415d7160ce1595e9a230301619d64] [ 1106.439864] ? auxdev_write_iter+0x4b0/0x4b0 [drm_display_helper d888144e430415d7160ce1595e9a230301619d64] [ 1106.439903] ? rcu_read_lock_sched_held+0x14/0x80 [ 1106.439916] vfs_read+0x578/0x810 [ 1106.439928] ? kernel_read+0x140/0x140 [ 1106.439945] ? __fget_light+0xbe/0x230 [ 1106.439959] ksys_read+0x10a/0x1e0 [ 1106.439969] ? __ia32_sys_pwrite64+0x1f0/0x1f0 [ 1106.439983] do_syscall_64+0x5f/0x90 [ 1106.439995] ? rcu_read_lock_sched_held+0x14/0x80 [ 1106.440004] ? trace_hardirqs_on_prepare+0x7c/0x160 [ 1106.440015] ? do_syscall_64+0x6b/0x90 [ 1106.440026] ? do_syscall_64+0x6b/0x90 [ 1106.440035] ? trace_hardirqs_on_prepare+0x7c/0x160 [ 1106.440046] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 1106.440056] RIP: 0033:0x7fa6d6bfc04c [ 1106.440064] RSP: 002b:00007ffe2324dc40 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [ 1106.440074] RAX: ffffffffffffffda RBX: 000055d8823ea020 RCX: 00007fa6d6bfc04c [ 1106.440082] RDX: 0000000000000001 RSI: 00007ffe2324dcec RDI: 000000000000000f [ 1106.440089] RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000040 [ 1106.440095] R10: 0000000000000000 R11: 0000000000000246 R12: 00000000000004b0 [ 1106.440101] R13: 0000000000000001 R14: 00007ffe2324dcec R15: 00007ffe2324dd70 [ 1106.440116] </TASK> [ 1106.440274] INFO: lockdep is turned off.Also, there's nothing on
/var/lib/systemd/pstore:[root@thinkryzen pstore]# pwd /var/lib/systemd/pstore [root@thinkryzen pstore]# ls -la total 8 drwxr-xr-x 2 root root 4096 Sep 30 09:15 . drwxr-xr-x 10 root root 4096 Oct 2 13:17 ..Is there any other options I can change to enable extra debug logs to help on this?
[ 1106.439271] INFO: task fwupd:2275 blocked for more than 122 seconds.I don't suspect fwupd is to blame here but rather is a victim. It's worth mentioning though it will probe over DP aux to look for synaptics MST hubs. At least until this other bug is sorted out I think it's worth masking or stopping that service to make sure it doesn't get involved with a race to make things more complicated.
-
OK - this is another bug but one I kinda had a feeling might happen (hence mentioning timeouts before). Knowing now that the
mutex_lock(&aconnector->mst_mgr.lock);line was broken, that'd imply that until now we haven't been acquiring &mst_mgr.lock at all - which means the way this code locks here might not even work in the real world and there's been a deadlock hiding here. I assumefwupddoes aux probing, and then the deadlock likely happens between whatever locking the block device uses and the MST topology lock.Could you recompile with this config option?
CONFIG_LOCKDEP_SUPPORT=yI expect that to tell us something more useful
Here's my lockdep configs (from the above logs):
# # Lock Debugging (spinlocks, mutexes, etc...) # CONFIG_LOCK_DEBUGGING_SUPPORT=y CONFIG_PROVE_LOCKING=y CONFIG_PROVE_RAW_LOCK_NESTING=y CONFIG_LOCK_STAT=y CONFIG_DEBUG_RT_MUTEXES=y CONFIG_DEBUG_SPINLOCK=y CONFIG_DEBUG_MUTEXES=y CONFIG_DEBUG_WW_MUTEX_SLOWPATH=y CONFIG_DEBUG_RWSEMS=y CONFIG_DEBUG_LOCK_ALLOC=y CONFIG_LOCKDEP=y CONFIG_LOCKDEP_BITS=15 CONFIG_LOCKDEP_CHAINS_BITS=16 CONFIG_LOCKDEP_STACK_TRACE_BITS=19 CONFIG_LOCKDEP_STACK_TRACE_HASH_BITS=14 CONFIG_LOCKDEP_CIRCULAR_QUEUE_BITS=12 CONFIG_DEBUG_LOCKDEP=y # CONFIG_DEBUG_ATOMIC_SLEEP is not set # CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set # CONFIG_LOCK_TORTURE_TEST is not set # CONFIG_WW_MUTEX_SELFTEST is not set # CONFIG_SCF_TORTURE_TEST is not set # CONFIG_CSD_LOCK_WAIT_DEBUG is not set # end of Lock Debugging (spinlocks, mutexes, etc...) # CONFIG_DEBUG_IRQFLAGS is not set CONFIG_STACKTRACE=y # CONFIG_WARN_ALL_UNSEEDED_RANDOM is not set # CONFIG_DEBUG_KOBJECT is not setCONFIG_LOCKDEP_SUPPORT=y CONFIG_STACKTRACE_SUPPORT=yEdited by Nicolas Frenay-
Also: feel free to turn KASAN off at this point, will likely make things go faster and also we might need it off anyway to stop another KASAN bug in amdgpu I'm seeing (use-after-free in
handle_hpd_irq_helper(), not sure it's related but it'll turn off lock debugging if you hit it).And finally: if you could run the
dmesgthrough the./scripts/decode_stacktrace.shscript in the kernel source repo that'd be super useful! You'll likely only be able to do this if you have access to the uncompressedvmlinuxfile from the kernel build. You can do it by running:$DMESG_CMD | ./scripts/decode_stacktrace.sh $VMLINUX auto $KMOD_DIRWhere
$DMESG_CMDis the command you're using to get the dmesg (dmesg,journalctl, etc.),$VMLINUXis the patch tovmlinux, and$KMOD_DIRis the directory with all of your.kofilesIf you can't get the last step working though feel free to post the dmesg anyway, running that script helps though since it'll give us actual line numbers from the kernel source instead of just symbol offsets
Here's that last log decoded. Never coded on the kernel, I didn't know the decode_stacktrace script :). Learned something new. lastcrash-decoded.txt
Will rebuild the kernel without KASAN now.
-
OK, I've pushed one more patch: lyudess/linux@04949338 could you give this a shot? (just apply the whole branch like last time)
Edited by Lyude Paul Just tried this last patch, and also disabled fwupd.service temporarily.
Good news: I'm not experiencing gpu crashes anymore. Bad news: when I plug both HDMI monitors on the USB dock I loose image on all 3 screens, but if I unplug one of them, they work again.
Following a sequence of events, it's a little more complicated:
- Booted with eDP + USB-Hub with 1 HDMI: video output on both monitors OK
- Plugged in additional HDMI monitor on USB-Hub: video output still working on the original 2 monitors, but not on the newly plugged one
- Removing HDMI and replugging them all gets to no video output on all monitors, but system does not crash. The order in which I plug them doesn't matter. As long as I have 3 monitors, they blackout, except from just after the initial boot, where when I plug the second HDMI, it just doesn't output on that one, but the other two keep working. Kind of weird.
On this playing around with different HDMI combos, I have the following log, with stacktraces: crash-v1c-decoded.txt
So, I think this is good progress, but there might be a catch somewhere. Might even be elsewhere.
In the part below, I don't remember seeing this
[drm] crtc[1] needs mode_changedlog before.[ 233.924007] [drm] DP Alt mode state on HPD: 1 [ 234.116868] Registered IR keymap rc-cec [ 234.117176] rc rc0: DP-2 as /devices/pci0000:00/0000:00:08.1/0000:33:00.0/rc/rc0 [ 234.117605] input: DP-2 as /devices/pci0000:00/0000:00:08.1/0000:33:00.0/rc/rc0/input57 [ 258.404001] [drm] DP Alt mode state on HPD: 1 [ 258.481702] [drm] DM_MST: starting TM on aconnector: 00000000d349c024 [id: 94] [ 258.648748] [drm] Downstream port present 1, type 2 [ 258.735750] [drm] Downstream port present 1, type 2 [ 258.866885] [drm] crtc[1] needs mode_changedThanks for all the effort. I can help with testing tomorrow.
-
Thank you for all of the testing! When you get to this tomorrow, it would be good if I could see a log of you reproducing this with
drm.debug=0x116 log_buf_len=50Madded on the kernel cmdline from the version of the kernel where this worked, and one from the current kernel. The whole kernel logs for each one as well. Here's what the top of tree looks like:
WIP: drm/amdgpu/dm/dp_mst: Don't grab mst_mgr->lock when computing DSC state drm/amdgpu/dm/mst: Use the correct topology mgr pointer in amdgpu_dm_connector drm/amd/dc/dce120: Fix audio register mapping, stop triggering KASAN drm/display/dp_mst: Fix drm_dp_mst_add_affected_dsc_crtcs() return code drm/amdgpu/mst: Stop ignoring error codes and deadlocking drm-tip: 2022y-11m-09d-17h-31m-37s UTC integration manifest@khfeng for the DPIA cases can you please roll these two patches into your series? https://patchwork.freedesktop.org/patch/510829/ https://patchwork.freedesktop.org/patch/510825/
-
@superm1 should I assume since you're posting the DPIA case related patches that what we're seeing now is a known issue?
KH was testing DPIA (DP tunneling) but NF was testing DP-Alt mode. There are two cases that DPIA has problems that those two patches help.
They're both going to submit into 6.1-fixes next week so it's best to do the test with them in place because you can end up with stack corruption and hangs potentially which could make the effort to affirm the fixups that you've developed infinitely more difficult.
-
Ahhh - I guess we'll have to wait for the feedback then from @nicolas.frenay
@superm1 The screen freeze issue remains with the two patches applied.
-
@khfeng Could you get a log from a kernel where this worked with the various kernel cmdline options added? Would like to see what we're doing differently that might be causing that atomic check failure
-
Oh nevermind lol, I think I already found the issue. Try again with lyudess/linux@f74340f6 applied :)
-
@nicolas.frenay I think this will fix your issue as well
@lyudess The result is positive, the kernel splat/screen freeze on hotplug is now fixed:
Tested-by: Kai-Heng Feng kai.heng.feng@canonical.com
However, the screen remains blank on system resume. The first error is from amdgpu side:
kernel: [drm:dc_dmub_srv_wait_idle [amdgpu]] *ERROR* Error waiting for DMUB idle: status=3So I think it has become a different issue now?
This was tested with drm-tip + lyudess/linux@f74340f6 + https://patchwork.freedesktop.org/patch/510829/ https://patchwork.freedesktop.org/patch/510825/
Log: journalctl
The result is positive, the kernel splat/screen freeze on hotplug is now fixed:
That's great!
So I think it has become a different issue now?
Maybe. I think need some more narrowing down.
No payload for [MST PORT:0000000098818ef5] found in mst state 0000000015e900f7
I think this is the original thing you were reporting about; and this rabbit hole of other bugs got intermingled.
However, the screen remains blank on system resume.
Internal or external screen remains blank? I noticed you did magic sysrq; so I guess internal screen?
kernel: [drm:dc_dmub_srv_wait_idle [amdgpu]] *ERROR* Error waiting for DMUB idle: status=3This specific message may be a red herring to the actual issue; but just the last visible thing you saw.
- Do you only see this message/behavior on suspend/resume with dock connected?
- If you unplug before suspend and plug in after suspend does everything work?
- Can that set of patches from Lyude + the two I pointed at reproduce this on top of latest 6.1-rc?
- A few other things you can try (independently of one another) to see if it helps on resume:
- Disable vblank delay:
drm.vblankoffdelay=0 - Disable PSR:
amdgpu.dcdebugmask=0x10
- Disable vblank delay:
added bug label
mentioned in issue #2181
mentioned in commit lyudess/linux@37bac505
mentioned in commit lyudess/linux@a28eb2b0
mentioned in commit lyudess/linux@b88b6f19
@lyudess just tested with the latest patch, and when I plug the second HDMI I get no output on both external monitors (using USB hub) but still have output on the eDP, with no crashes.
Interestingly, if I open gnome's display settings, I can see all 3 displays, even though I only have output on the laptop LCD.
The error is now on a different part of the code:
[ 55.941888] WARNING: CPU: 12 PID: 1170 at drivers/gpu/drm/amd/amdgpu/../display/dc/core/dc_link.c:3533 update_mst_stream_alloc_table (/home/nicolas/aur/linux-mainline/src/linux-mainline/drivers/gpu/drm/amd/amdgpu/../display/dc/core/dc_link.c:3533 (discriminator 1)) amdgpuHere's the decoded logs: no-output-decoded.txt
The first errors on the log happen when I plug the second HDMI as usual. The last error (at around 373.643691) happen when I unplug this monitor (did this for testing, just in case it helps).
Unplugging the HDMI2 gives me back image on eDP + HDMI1 external via usb.
I couldn't test yesterday, sorry about that. Also, I don't have a successful reproducer kernel version, as this is a new laptop and I was running kernel 6.x since day one. I can try using an older kernel, but not sure if that would work.
-
@nicolas.frenay @khfeng could either one of you get the same logs but with
drm.debug=0x16 log_buf_len=50Madded as well to the kernel commandline? It's possible this is still an issue with the MST helpers
