regression: VRAM stuck at minimum Hz (Hawaii PRO - AMDGPU)

Brief summary of the problem:

With recent kernels running AMDGPU the memory on my R9 390 is locked to 150MHz

'cat /sys/class/drm/card1/device/pp_dpm_mclk' shows that 150MHz and 1500MHz are still available but running games, forcing perf modes etc does not effect mclk at all while GPU core clock scales as normal. game performance is a fraction of what it should be as a result of this.

the Radeon driver is not affected by this issue, it only occurs with AMDGPU on 6.13+

6.14.0-rc4-1-mainline  BAD
6.13.4-zen1-1-zen      BAD
6.13.4-arch1-1         BAD
6.13.1-arch1-1         BAD
6.13.0-arch1-1         BAD
6.12.16-1-lts          GOOD
6.12.12-hardened1-2-   GOOD
6.12.10-arch1-1        GOOD
6.10.2-rt14-arch1-5-rt GOOD

testing was done on 2025-02-22 with my main Arch OS, but this issue persists on a clean CachyOS/Arch install running linux 6.13.x

dmesg shows these two errors only with 6.13+

[    7.406303] [drm:amdgpu_device_ip_set_powergating_state [amdgpu]] *ERROR* set_powergating_state of IP block <vce_v2_0> failed -22
[   23.149934] amdgpu 0000:05:00.0: amdgpu: failed to send message 146 ret is 0]

and hwinfo on 6.13+

  <6>[ 4996.475597] amdgpu 0000:05:00.0: amdgpu: failed to send message 18b ret is 0
  <6>[ 4996.475614] amdgpu 0000:05:00.0: amdgpu: failed to send message 18c ret is 255
  <6>[ 4998.654332] amdgpu 0000:05:00.0: amdgpu: failed to send message 146 ret is 0
  <6>[ 5246.115943] amdgpu 0000:05:00.0: amdgpu: failed to send message 18c ret is 255

Hardware description:

CPU: AMD 3500x (zen 2) GPU: 05:00.0 VGA compatible controller [0300]: [AMD/ATI] Hawaii PRO [Radeon R9 290/390] [1002:67b1] (rev 80) System Memory: 16GB Type of Display Connection: DP

System information:

Distro name and Version: Arch rolling, all up-to-date stable

enabling and disabling resizable BAR, 4G decode in bios and the overclock boot flag have no effect. dmesg_out_6-14-rc4

not sure what else I can add, I'm pretty clueless about bisecting and patching kernels.

I think this patch should fix it: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=960a62877466067adc89bd37fe36d3b6edddb965

would that patch have been included in the chaotic-AUR build of 6.14rc5? the issue persists with that kernel, and I just noticed that GPU voltage bottoms out at 1087mV instead of 900mV, causing an 8-10W increase in idle power draw. still getting [drm:amdgpu_device_ip_set_powergating_state [amdgpu]] *ERROR* set_powergating_state of IP block <vce_v2_0> failed -22 but it looks like the other errors are gone.

sudo dmesg | grep amdgpu
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=3cf89746-f4af-4b23-a63e-6fd0ea2dcff6 rw quiet loglevel=2 nohpet selinux=0 nowatchdog module_blacklist=pcspkr zswap.enabled=0 radeon.cik_support=0 radeon.si_support=0 amdgpu.cik_support=1 amdgpu.si_support=1 amdgpu.dc=1 amdgpu.dpm=1
[    0.029057] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=3cf89746-f4af-4b23-a63e-6fd0ea2dcff6 rw quiet loglevel=2 nohpet selinux=0 nowatchdog module_blacklist=pcspkr zswap.enabled=0 radeon.cik_support=0 radeon.si_support=0 amdgpu.cik_support=1 amdgpu.si_support=1 amdgpu.dc=1 amdgpu.dpm=1
[    5.602118] [drm] amdgpu kernel modesetting enabled.
[    5.607255] amdgpu: Virtual CRAT table created for CPU
[    5.607264] amdgpu: Topology: Add CPU node
[    5.607327] amdgpu 0000:05:00.0: enabling device (0006 -> 0007)
[    5.607437] amdgpu 0000:05:00.0: amdgpu: detected ip block number 0 <cik_common>
[    5.607440] amdgpu 0000:05:00.0: amdgpu: detected ip block number 1 <gmc_v7_0>
[    5.607442] amdgpu 0000:05:00.0: amdgpu: detected ip block number 2 <cik_ih>
[    5.607443] amdgpu 0000:05:00.0: amdgpu: detected ip block number 3 <gfx_v7_0>
[    5.607444] amdgpu 0000:05:00.0: amdgpu: detected ip block number 4 <cik_sdma>
[    5.607445] amdgpu 0000:05:00.0: amdgpu: detected ip block number 5 <powerplay>
[    5.607446] amdgpu 0000:05:00.0: amdgpu: detected ip block number 6 <dm>
[    5.607448] amdgpu 0000:05:00.0: amdgpu: detected ip block number 7 <uvd_v4_2>
[    5.607449] amdgpu 0000:05:00.0: amdgpu: detected ip block number 8 <vce_v2_0>
[    5.607462] amdgpu 0000:05:00.0: amdgpu: Fetched VBIOS from VFCT
[    5.607464] amdgpu: ATOM BIOS: 113-GRENADA_PRO_C671_D5_8GB_HY_OC_W8 
[    5.607552] amdgpu: KFD support on Hawaii is experimental. See modparam exp_hw_support
[    5.607554] kfd kfd: amdgpu: HAWAII  not supported in kfd
[    5.638454] amdgpu 0000:05:00.0: vgaarb: deactivate vga console
[    5.638458] amdgpu 0000:05:00.0: amdgpu: Trusted Memory Zone (TMZ) feature not supported
[    5.638738] amdgpu 0000:05:00.0: amdgpu: VRAM: 8192M 0x000000F400000000 - 0x000000F5FFFFFFFF (8192M used)
[    5.638741] amdgpu 0000:05:00.0: amdgpu: GART: 1024M 0x000000FF00000000 - 0x000000FF3FFFFFFF
[    5.638820] [drm] amdgpu: 8192M of VRAM memory ready
[    5.638821] [drm] amdgpu: 7948M of GTT memory ready.
[    5.641640] amdgpu: hwmgr_sw_init smu backed is ci_smu
[    5.674270] amdgpu 0000:05:00.0: [drm] dce110_link_encoder_construct: Failed to get encoder_cap_info from VBIOS with error code 4!
[    5.674287] amdgpu 0000:05:00.0: [drm] dce110_link_encoder_construct: Failed to get encoder_cap_info from VBIOS with error code 4!
[    5.674303] amdgpu 0000:05:00.0: [drm] dce110_link_encoder_construct: Failed to get encoder_cap_info from VBIOS with error code 4!
[    5.899071] amdgpu 0000:05:00.0: amdgpu: SE 4, SH per SE 1, CU per SH 11, active_cu_number 40
[    6.407005] amdgpu 0000:05:00.0: amdgpu: Using BACO for runtime pm
[    6.407407] amdgpu 0000:05:00.0: [drm] Registered 6 planes with drm panic
[    6.407409] [drm] Initialized amdgpu 3.61.0 for 0000:05:00.0 on minor 1
[    6.411443] fbcon: amdgpudrmfb (fb0) is primary device
[    6.411446] amdgpu 0000:05:00.0: [drm] fb0: amdgpudrmfb frame buffer device
[    7.470238] [drm:amdgpu_device_ip_set_powergating_state [amdgpu]] *ERROR* set_powergating_state of IP block <vce_v2_0> failed -22
[    7.890461] snd_hda_intel 0000:05:00.1: bound 0000:05:00.0 (ops amdgpu_dm_audio_component_bind_ops [amdgpu])
[   13.207211] amdgpu 0000:05:00.0: amdgpu: Disabling VM faults because of PRT request!

If that patch doesn't fix it, can you bisect?

I had this bug too and managed to seemingly resolve it. At least both the error about powergating and the stuck memory clock frequency are gone now.

The cause is likely a typo in file drivers/gpu/drm/amd/amdgpu/vce_v2_0.c, function vce_v2_0_stop:

...
ip_block = amdgpu_device_ip_get_ip_block(adev, AMD_IP_BLOCK_TYPE_VCN);
if (!ip_block)
    return -EINVAL;
...

Note the enum value AMD_IP_BLOCK_TYPE_VCN, which should probably be AMD_IP_BLOCK_TYPE_VCE.

Good catch. The attached patch should fix it.

0001-drm-amdgpu-vce2-fix-ip-block-reference.patch

thanks @selendym that must have been it. @agd5f looks like that patch fixes it. mclk and voltage are now behaving as they should on what uname -r describes as 6.14.0-rc6-1-git-dirty I tried to bisect yesterday but ran into problems booting the older kernel versions for reasons still uncertain to me.

now the only errors I see in dmesg are

[    5.628723] amdgpu 0000:05:00.0: [drm] dce110_link_encoder_construct: Failed to get encoder_cap_info from VBIOS with error code 4!
[    5.628741] amdgpu 0000:05:00.0: [drm] dce110_link_encoder_construct: Failed to get encoder_cap_info from VBIOS with error code 4!
[    5.628759] amdgpu 0000:05:00.0: [drm] dce110_link_encoder_construct: Failed to get encoder_cap_info from VBIOS with error code 4!

but that might be due to an unrelated ROCm limitation/lack of support.

closed

mentioned in commit agd5f/linux@02438acd

mentioned in commit agd5f/linux@ded6ad4c

mentioned in commit nouveau@d54e7235