[regression] amdgpu causes kernel crash on linux 6.7.2 / Lenovo Z16 / Ryzen 7 PRO 6850H iGPU
Brief summary of the problem:
Since some point in the Linux 6.6 series of kernels, I started having issues booting my Lenovo Z16 (specs below) when using the integrated GPU. Booting on the hybrid GPU works without issue. In particular, when booting, the normal list of services that systemd starts at boot freezes towards the end of the sequence, always on the Realtime PolicyKit Service message.
The system is up in some senses, in that services are running and writing to the journal (and I could SSH into the machine if I wanted), but my desktop environment won't load and I can't get to a tty.
Since I can load the latest kernel by using the hybrid GPU, or use the iGPU on the LTS kernel, this isn't ending my world. But I hadn't seen this issue reported yet anywhere, so I thought I'd take a stab at my first-ever bug report and kernel bisect. I have done my best to provide the relevant information, but am happy to provide more/different information if I missed the mark.
Hardware description:
- CPU: Ryzen 7 PRO 6850H
- GPU: Radeon 680M
- System Memory: 16GB
- Display(s): 1
- Type of Display Connection: eDP (integrated display)
System information:
- Distro name and Version: Arch Linux
- Kernel version: 6.7.2
- Custom kernel: n/a; have bisected back to 6.6
- AMD official driver version: n/a
How to reproduce the issue:
Install stock Linux mainline kernel from Arch repos, ensure you have set the machine to utilize the integrated GPU, boot.
Attached files:
Characteristic boot log that I get when this issue manifests
Note: this one was a random one I saved when bisecting the kernel, but it looks just the same on 6.7.2 and each earlier version.
Jan 30 09:41:51 archlinux kernel: Linux version 6.7.0-rc6-1-git-00020-g9c749e61a110 (linux-git@archlinux) (gcc (GCC) 13.2.1 20230801, GNU ld (GNU Binutils) 2.41.0) #10 SMP PREEMPT_DYNAMIC Tue, 30 Jan 2024 13:54:27 +0000
Jan 30 09:41:51 archlinux kernel: Command line: i8042.direct i8042.dumbkbd rd.luks.name=4e7e0157-6216-4a6a-88ab-d962a9292024=luksroot nowatchdog amd_pstate=active zswap.enabled=0 root=/dev/mapper/luksroot rw rootfstype=f2fs lockdown=integrity module.sig_enforce=1 mem_encrypt=on
[ * * * ]
Jan 30 09:42:18 fossbookpro kernel: [drm] PCIE GART of 512M enabled (table at 0x00000080FEB00000).
Jan 30 09:42:18 fossbookpro kernel: [drm] PSP is resuming...
Jan 30 09:42:18 fossbookpro kernel: [drm] reserve 0xa00000 from 0x80fd000000 for PSP TMR
Jan 30 09:42:19 fossbookpro kernel: amdgpu 0000:03:00.0: amdgpu: RAS: optional ras ta ucode is not available
Jan 30 09:42:19 fossbookpro kernel: amdgpu 0000:03:00.0: amdgpu: SECUREDISPLAY: securedisplay ta ucode is not available
Jan 30 09:42:19 fossbookpro kernel: amdgpu 0000:03:00.0: amdgpu: SMU is resuming...
Jan 30 09:42:19 fossbookpro kernel: amdgpu 0000:03:00.0: amdgpu: smu driver if version = 0x0000000d, smu fw if version = 0x00000010, smu fw program = 0, version = 0x00492200 (73.34.0)
Jan 30 09:42:19 fossbookpro kernel: amdgpu 0000:03:00.0: amdgpu: SMU driver if version not matched
Jan 30 09:42:19 fossbookpro kernel: amdgpu 0000:03:00.0: amdgpu: SMU is resumed successfully!
Jan 30 09:42:19 fossbookpro kernel: [drm] DMUB hardware initialized: version=0x02020020
Jan 30 09:42:19 fossbookpro kernel: [drm] kiq ring mec 2 pipe 1 q 0
Jan 30 09:42:19 fossbookpro kernel: [drm] VCN decode and encode initialized successfully(under DPG Mode).
Jan 30 09:42:19 fossbookpro kernel: amdgpu 0000:03:00.0: amdgpu: ring gfx_0.0.0 uses VM inv eng 0 on hub 0
Jan 30 09:42:19 fossbookpro kernel: amdgpu 0000:03:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 1 on hub 0
Jan 30 09:42:19 fossbookpro kernel: amdgpu 0000:03:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 4 on hub 0
Jan 30 09:42:19 fossbookpro kernel: amdgpu 0000:03:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 5 on hub 0
Jan 30 09:42:19 fossbookpro kernel: amdgpu 0000:03:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 6 on hub 0
Jan 30 09:42:19 fossbookpro kernel: amdgpu 0000:03:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 7 on hub 0
Jan 30 09:42:19 fossbookpro kernel: amdgpu 0000:03:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 8 on hub 0
Jan 30 09:42:19 fossbookpro kernel: amdgpu 0000:03:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 9 on hub 0
Jan 30 09:42:19 fossbookpro kernel: amdgpu 0000:03:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 10 on hub 0
Jan 30 09:42:19 fossbookpro kernel: amdgpu 0000:03:00.0: amdgpu: ring kiq_0.2.1.0 uses VM inv eng 11 on hub 0
Jan 30 09:42:19 fossbookpro kernel: amdgpu 0000:03:00.0: amdgpu: ring sdma0 uses VM inv eng 12 on hub 0
Jan 30 09:42:19 fossbookpro kernel: amdgpu 0000:03:00.0: amdgpu: ring vcn_dec_0 uses VM inv eng 0 on hub 8
Jan 30 09:42:19 fossbookpro kernel: BUG: kernel NULL pointer dereference, address: 00000000000001b8
Jan 30 09:42:19 fossbookpro kernel: #PF: supervisor read access in kernel mode
Jan 30 09:42:19 fossbookpro kernel: #PF: error_code(0x0000) - not-present page
Jan 30 09:42:19 fossbookpro kernel: PGD 0 P4D 0
Jan 30 09:42:19 fossbookpro kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI
Jan 30 09:42:19 fossbookpro kernel: CPU: 10 PID: 1407 Comm: systemd-logind Not tainted 6.7.0-rc6-1-git-00020-g9c749e61a110 #10 8d1c80d091417e1f94d2b66ef25a7cce8a70f01f
Jan 30 09:42:19 fossbookpro kernel: Hardware name: LENOVO 21D4000GUS/21D4000GUS, BIOS N3GET64W (1.64 ) 11/01/2023
Jan 30 09:42:19 fossbookpro kernel: RIP: 0010:amdgpu_vm_init+0x2ac/0x4d0 [amdgpu]
Jan 30 09:42:19 fossbookpro kernel: Code: 00 00 48 8b b8 50 01 00 00 e8 70 83 5b e2 89 c5 85 c0 0f 84 1f 01 00 00 48 89 de 4c 89 e7 e8 5b 28 00 00 4c 8b a3 e8 00 00 00 <49> 8b 84 24 b8 01 00 00 48 8d b8 20 08 00 00 e8 00 d7 9d e2 49 8d
Jan 30 09:42:19 fossbookpro kernel: RSP: 0018:ffffc90004bdba60 EFLAGS: 00010286
Jan 30 09:42:19 fossbookpro kernel: RAX: 0000000000000000 RBX: ffff88810793d000 RCX: ffff88810793d070
Jan 30 09:42:19 fossbookpro kernel: RDX: ffff88813e7b81d0 RSI: ffffc90004bdba00 RDI: ffff888114080000
Jan 30 09:42:19 fossbookpro kernel: RBP: 00000000ffffffed R08: ffffffffc051c448 R09: 000000000000001b
Jan 30 09:42:19 fossbookpro kernel: R10: ffffffffc051c440 R11: 0000000000000000 R12: 0000000000000000
Jan 30 09:42:19 fossbookpro kernel: R13: ffff88813e7b8000 R14: ffff88811408ef40 R15: ffff888114080000
Jan 30 09:42:19 fossbookpro kernel: FS: 00007fe6c4d1cf00(0000) GS:ffff88844e680000(0000) knlGS:0000000000000000
Jan 30 09:42:19 fossbookpro kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 30 09:42:19 fossbookpro kernel: CR2: 00000000000001b8 CR3: 0000000135fce000 CR4: 0000000000f50ef0
Jan 30 09:42:19 fossbookpro kernel: PKRU: 55555554
Jan 30 09:42:19 fossbookpro kernel: Call Trace:
Jan 30 09:42:19 fossbookpro kernel: <TASK>
Jan 30 09:42:19 fossbookpro kernel: ? __die+0x23/0x70
Jan 30 09:42:19 fossbookpro kernel: ? page_fault_oops+0x171/0x4e0
Jan 30 09:42:19 fossbookpro kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Jan 30 09:42:19 fossbookpro kernel: ? exc_page_fault+0x7f/0x180
Jan 30 09:42:19 fossbookpro kernel: ? asm_exc_page_fault+0x26/0x30
Jan 30 09:42:19 fossbookpro kernel: ? amdgpu_vm_init+0x2ac/0x4d0 [amdgpu cef1599d786919d70326b31149436a8677985f4e]
Jan 30 09:42:19 fossbookpro kernel: amdgpu_driver_open_kms+0x106/0x270 [amdgpu cef1599d786919d70326b31149436a8677985f4e]
Jan 30 09:42:19 fossbookpro kernel: drm_file_alloc+0x1b7/0x260
Jan 30 09:42:19 fossbookpro kernel: drm_open_helper+0x7e/0x150
Jan 30 09:42:19 fossbookpro kernel: drm_open+0x7f/0x140
Jan 30 09:42:19 fossbookpro kernel: drm_stub_open+0xae/0xe0
Jan 30 09:42:19 fossbookpro kernel: chrdev_open+0xd1/0x250
Jan 30 09:42:19 fossbookpro kernel: ? __pfx_chrdev_open+0x10/0x10
Jan 30 09:42:19 fossbookpro kernel: do_dentry_open+0x203/0x590
Jan 30 09:42:19 fossbookpro kernel: path_openat+0xd7d/0x1260
Jan 30 09:42:19 fossbookpro kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Jan 30 09:42:19 fossbookpro kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Jan 30 09:42:19 fossbookpro kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Jan 30 09:42:19 fossbookpro kernel: ? dput+0x3a/0x310
Jan 30 09:42:19 fossbookpro kernel: do_filp_open+0xb3/0x160
Jan 30 09:42:19 fossbookpro kernel: do_sys_openat2+0xab/0xe0
Jan 30 09:42:19 fossbookpro kernel: __x64_sys_openat+0x57/0xa0
Jan 30 09:42:19 fossbookpro kernel: do_syscall_64+0x64/0xe0
Jan 30 09:42:19 fossbookpro kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Jan 30 09:42:19 fossbookpro kernel: ? do_readlinkat+0x109/0x140
Jan 30 09:42:19 fossbookpro kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Jan 30 09:42:19 fossbookpro kernel: ? syscall_exit_to_user_mode+0x2b/0x40
Jan 30 09:42:19 fossbookpro kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Jan 30 09:42:19 fossbookpro kernel: ? do_syscall_64+0x70/0xe0
Jan 30 09:42:19 fossbookpro kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Jan 30 09:42:19 fossbookpro kernel: ? syscall_exit_to_user_mode+0x2b/0x40
Jan 30 09:42:19 fossbookpro kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Jan 30 09:42:19 fossbookpro kernel: ? do_syscall_64+0x70/0xe0
Jan 30 09:42:19 fossbookpro kernel: entry_SYSCALL_64_after_hwframe+0x6e/0x76
Jan 30 09:42:19 fossbookpro kernel: RIP: 0033:0x7fe6c584cbf2
Jan 30 09:42:19 fossbookpro kernel: Code: 83 e2 40 75 53 89 f0 f7 d0 a9 00 00 41 00 74 48 80 3d 81 d7 0d 00 00 74 6c 89 da 48 89 ee bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 92 00 00 00 48 8b 54 24 28 64 48 2b 14 25
Jan 30 09:42:19 fossbookpro kernel: RSP: 002b:00007ffc7b0c79b0 EFLAGS: 00000202 ORIG_RAX: 0000000000000101
Jan 30 09:42:19 fossbookpro kernel: RAX: ffffffffffffffda RBX: 0000000000080902 RCX: 00007fe6c584cbf2
Jan 30 09:42:19 fossbookpro kernel: RDX: 0000000000080902 RSI: 000055da6529ba60 RDI: 00000000ffffff9c
Jan 30 09:42:19 fossbookpro kernel: RBP: 000055da6529ba60 R08: 0000000000000020 R09: 000055da6529c848
Jan 30 09:42:19 fossbookpro kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000001
Jan 30 09:42:19 fossbookpro kernel: R13: 00007ffc7b0c7af0 R14: 0000000000000008 R15: 0000000000000001
Jan 30 09:42:19 fossbookpro kernel: </TASK>
Jan 30 09:42:19 fossbookpro kernel: Modules linked in: rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device ccm algif_aead crypto_null des3_ede_x86_64 des_generic libdes md4 cmac algif_hash algif_skcipher af_alg bnep qrtr_mhi ip6table_filter ip6_tables iptable_filter snd_ctl_led uvcvideo videobuf2_vmalloc uvc btusb videobuf2_memops btrtl videobuf2_v4l2 btintel btbcm videodev btmtk videobuf2_common bluetooth mc ecdh_generic crc16 intel_rapl_msr intel_rapl_common snd_soc_dmic snd_soc_acp6x_mach snd_acp6x_pdm_dma snd_sof_amd_vangogh snd_sof_amd_rembrandt snd_sof_amd_renoir snd_sof_amd_acp snd_sof_pci snd_sof_xtensa_dsp qrtr snd_sof ath11k_pci snd_sof_utils edac_mce_amd snd_soc_core ath11k snd_compress kvm_amd snd_hda_codec_realtek joydev vfat qmi_helpers ac97_bus snd_hda_scodec_cs35l41_spi mousedev fat snd_hda_codec_generic snd_pcm_dmaengine snd_hda_codec_hdmi snd_pci_ps snd_rpl_pci_acp6x kvm mac80211 snd_acp_pci snd_hda_scodec_cs35l41_i2c snd_hda_intel snd_acp_legacy_common snd_hda_scodec_cs35l41 snd_intel_dspcfg hid_multitouch ucsi_acpi
Jan 30 09:42:19 fossbookpro kernel: irqbypass snd_hda_cs_dsp_ctls snd_intel_sdw_acpi snd_pci_acp6x snd_hda_codec cs_dsp libarc4 typec_ucsi snd_pci_acp5x think_lmi(+) i2c_hid_acpi snd_soc_cs35l41_lib typec rapl i2c_hid firmware_attributes_class wmi_bmof amd_pmc snd_hda_core snd_rn_pci_acp3x roles pcspkr amd_pmf sp5100_tco snd_hwdep cfg80211 serial_multi_instantiate snd_acp_config acpi_tad thunderbolt snd_pcm snd_soc_acpi i2c_piix4 snd_pci_acp3x mhi snd_timer mac_hid psmouse tcp_bbr sch_cake thinkpad_acpi ledtrig_audio platform_profile rfkill snd soundcore pkcs8_key_parser sg crypto_user fuse loop nfnetlink zram ip_tables x_tables f2fs crc32_generic lz4hc_compress lz4_compress dm_crypt cbc encrypted_keys trusted asn1_encoder tee amdgpu dm_mod crct10dif_pclmul crc32_pclmul crc32c_intel amdxcp polyval_clmulni i2c_algo_bit polyval_generic drm_ttm_helper gf128mul ttm ghash_clmulni_intel sha512_ssse3 drm_exec serio_raw atkbd sha256_ssse3 gpu_sched libps2 sha1_ssse3 drm_suballoc_helper sdhci_pci aesni_intel vivaldi_fmap cqhci drm_buddy nvme sdhci
Jan 30 09:42:19 fossbookpro kernel: crypto_simd drm_display_helper cryptd xhci_pci mmc_core nvme_core ccp xhci_pci_renesas cec video i8042 serio wmi
Jan 30 09:42:19 fossbookpro kernel: CR2: 00000000000001b8
Jan 30 09:42:19 fossbookpro kernel: ---[ end trace 0000000000000000 ]---
Jan 30 09:42:19 fossbookpro kernel: RIP: 0010:amdgpu_vm_init+0x2ac/0x4d0 [amdgpu]
Jan 30 09:42:19 fossbookpro kernel: Code: 00 00 48 8b b8 50 01 00 00 e8 70 83 5b e2 89 c5 85 c0 0f 84 1f 01 00 00 48 89 de 4c 89 e7 e8 5b 28 00 00 4c 8b a3 e8 00 00 00 <49> 8b 84 24 b8 01 00 00 48 8d b8 20 08 00 00 e8 00 d7 9d e2 49 8d
Jan 30 09:42:19 fossbookpro kernel: RSP: 0018:ffffc90004bdba60 EFLAGS: 00010286
Jan 30 09:42:19 fossbookpro kernel: RAX: 0000000000000000 RBX: ffff88810793d000 RCX: ffff88810793d070
Jan 30 09:42:19 fossbookpro kernel: RDX: ffff88813e7b81d0 RSI: ffffc90004bdba00 RDI: ffff888114080000
Jan 30 09:42:19 fossbookpro kernel: RBP: 00000000ffffffed R08: ffffffffc051c448 R09: 000000000000001b
Jan 30 09:42:19 fossbookpro kernel: R10: ffffffffc051c440 R11: 0000000000000000 R12: 0000000000000000
Jan 30 09:42:19 fossbookpro kernel: R13: ffff88813e7b8000 R14: ffff88811408ef40 R15: ffff888114080000
Jan 30 09:42:19 fossbookpro kernel: FS: 00007fe6c4d1cf00(0000) GS:ffff88844e680000(0000) knlGS:0000000000000000
Jan 30 09:42:19 fossbookpro kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 30 09:42:19 fossbookpro kernel: CR2: 00000000000001b8 CR3: 0000000135fce000 CR4: 0000000000f50ef0
Jan 30 09:42:19 fossbookpro kernel: PKRU: 55555554
Jan 30 09:42:19 fossbookpro kernel: note: systemd-logind[1407] exited with irqs disabled
Jan 30 09:42:19 fossbookpro kernel: amdgpu 0000:03:00.0: amdgpu: amdgpu: finishing device.
Jan 30 09:42:19 fossbookpro systemd[1]: Startup finished in 14.728s (firmware) + 2.284s (loader) + 708ms (kernel) + 23.103s (initrd) + 4.861s (userspace) = 45.687s.Kernel bisect results
Note: I don't understand why it would identify this commit; makes me think I didn't do the bisect correctly.
9c749e61a1104ba9cf00519e723a7fa8ac55db0d is the first bad commit
commit 9c749e61a1104ba9cf00519e723a7fa8ac55db0d
Merge: 3f10e214a9de 9a733dc4fbee
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date: Tue Dec 19 12:19:25 2023 -0800
Merge tag 'arc-6.7-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/vgupta/arc
Pull ARC fixes from Vineet Gupta:
- build error for hugetlb, sparse and smatch fixes
- removal of VIPT aliasing cache code
* tag 'arc-6.7-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/vgupta/arc:
ARC: add hugetlb definitions
ARC: fix smatch warning
ARC: fix spare error
ARC: mm: retire support for aliasing VIPT D$
ARC: entry: move ARCompact specific bits out of entry.h
ARC: entry: SAVE_ABI_CALLEE_REG: ISA/ABI specific helper
arch/arc/Kconfig | 5 --
arch/arc/include/asm/cacheflush.h | 43 -----------
arch/arc/include/asm/entry-arcv2.h | 32 +++++++++
arch/arc/include/asm/entry-compact.h | 87 +++++++++++++++++++++-
arch/arc/include/asm/entry.h | 110 ++--------------------------
arch/arc/include/asm/hugepage.h | 7 ++
arch/arc/include/asm/ptrace.h | 14 ++--
arch/arc/kernel/setup.c | 4 +-
arch/arc/kernel/signal.c | 6 +-
arch/arc/mm/cache.c | 136 ++---------------------------------
arch/arc/mm/mmap.c | 21 +-----
arch/arc/mm/tlb.c | 16 ++---
12 files changed, 155 insertions(+), 326 deletions(-)Kernel bisect log
git bisect start
# status: waiting for both good and bad commits
# bad: [0dd3ee31125508cd67f7e7172247f05b7fd1753a] Linux 6.7
git bisect bad 0dd3ee31125508cd67f7e7172247f05b7fd1753a
# status: waiting for good commit(s), bad commit known
# good: [ffc253263a1375a65fa6c9f62a893e9767fbebfa] Linux 6.6
git bisect good ffc253263a1375a65fa6c9f62a893e9767fbebfa
# good: [deefd5024f0772cf56052ace9a8c347dc70bcaf3] Merge tag 'vfio-v6.7-rc1' of https://github.com/awilliam/linux-vfio
git bisect good deefd5024f0772cf56052ace9a8c347dc70bcaf3
# good: [1f24458a1071f006e3f7449c08ae0f12af493923] Merge tag 'tty-6.7-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty
git bisect good 1f24458a1071f006e3f7449c08ae0f12af493923
# good: [ac347a0655dbc7d885e217c89dddd16e2800bd58] Merge tag 'arm64-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
git bisect good ac347a0655dbc7d885e217c89dddd16e2800bd58
# good: [c1bacb02cb2f2c56db10f3d927acb5f91fe4d064] MAINTAINERS: change vmware.com addresses to broadcom.com
git bisect good c1bacb02cb2f2c56db10f3d927acb5f91fe4d064
# good: [01a564bab4876007ce35f312e16797dfe40e4823] net: check vlan filter feature in vlan_vids_add_by_dev() and vlan_vids_del_by_dev()
git bisect good 01a564bab4876007ce35f312e16797dfe40e4823
# bad: [2e957f9c32c154f8b823caa4f0921e0056c0c253] Merge branch '40GbE' of git://git.kernel.org/pub/scm/linux/kernel/git/tnguy/net-queue
git bisect bad 2e957f9c32c154f8b823caa4f0921e0056c0c253
# bad: [9c749e61a1104ba9cf00519e723a7fa8ac55db0d] Merge tag 'arc-6.7-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/vgupta/arc
git bisect bad 9c749e61a1104ba9cf00519e723a7fa8ac55db0d
# good: [c8e97fc6b4c057a350a9e9a1ad625e10cc9c39ee] Merge tag 'arm64-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
git bisect good c8e97fc6b4c057a350a9e9a1ad625e10cc9c39ee
# good: [134fdb80bc130dba429295ac64358b16b695628c] Merge tag 'cxl-fixes-6.7-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl
git bisect good 134fdb80bc130dba429295ac64358b16b695628c
# good: [177c2ffe69555dde28fad5ddb62a6d806982e53f] Merge tag 'perf_urgent_for_v6.7_rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good 177c2ffe69555dde28fad5ddb62a6d806982e53f
# good: [2cf4f94d8e8646803f8fb0facf134b0cd7fb691a] Merge tag 'scsi-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi
git bisect good 2cf4f94d8e8646803f8fb0facf134b0cd7fb691a
# good: [4eb69d00fe967699b9d93f7e74a990fe813e8d2b] ARC: fix smatch warning
git bisect good 4eb69d00fe967699b9d93f7e74a990fe813e8d2b
# good: [6eb04ca8c52e3f8c8ea7102ade81d642eee87f4a] HID: nintendo: Prevent divide-by-zero on code
git bisect good 6eb04ca8c52e3f8c8ea7102ade81d642eee87f4a
# good: [9a733dc4fbeec3f6d99645b845712b035e7440cf] ARC: add hugetlb definitions
git bisect good 9a733dc4fbeec3f6d99645b845712b035e7440cf
# good: [3f10e214a9de738832b357a58b605c2fbd23aa96] Merge tag 'hid-for-linus-2023121901' of git://git.kernel.org/pub/scm/linux/kernel/git/hid/hid
git bisect good 3f10e214a9de738832b357a58b605c2fbd23aa96
# first bad commit: [9c749e61a1104ba9cf00519e723a7fa8ac55db0d] Merge tag 'arc-6.7-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/vgupta/arcEdited by Ian Foss