Due to an influx of spam, we have had to impose restrictions on new accounts. Please see this wiki page for instructions on how to get full permissions. Sorry for the inconvenience.
Admin message
Equinix is shutting down its operations with us on April 30, 2025. They have graciously supported us for almost 5 years, but all good things come to an end. We are expecting to transition to new infrastructure between late March and mid-April. We do not yet have a firm timeline for this, but it will involve (probably multiple) periods of downtime as we move our services whilst also changing them to be faster and more responsive. Any updates will be posted in freedesktop/freedesktop#2011 as it becomes clear, and any downtime will be announced with further broadcast messages.
Massive performance regression in Dota 2 (maybe others) on Linux 6.10.2 compared to Linux-LTS 6.6.42
Performance in dota 2 (and maybe others, see benchmarks below) seems to have regressed substantially (~82% performance loss) in kernels newer than linux-lts.
(if you are unable to access the links above I can provide the data elsewhere).
How to reproduce the issue:
Download Dota 2 and launch any in-game rendering on Linux 6.10.2
Extra information:
Below I have benchmarked Dota 2 (linux native vulkan build), Rise of the Shadow of the Tomb Raider (linux native build), Cyberpunk 2077 (Proton 9.0-2) and Geekbench6 single/multicore, with the aim of finding out if it is gpu or cpu issue, and a variety of graphics APIs.
The benchmarks are as follows (with settings for reproduction):
Dota 2:
Settings and camera position: https://imgur.com/a/2hmuYFi
I selected the hero Ancient Apparition (issue occurs with all heroes), selected "Demo Hero", and did not move the camera.
kernel
fps
linux-lts
490-530
linux
80-100
Cyberpunk 2077:
I selected the Ultra preset, no raytracing, and ensured it was set to 2560x1440 with vsync disabled, and used the built-in benchmark.
kernel
run 1
run 2
run 3
average
linux-lts
157.36
156.69
157.11
157.05
linux
156.00
156.69
155.68
156.12
Rise of the Tomb Raider:
I selected the Very High preset, set anti-aliasing to SMAA, ensured it was set to 2560x1440 with vsync disabled, and used the built-in benchmark.
kernel
run 1
run 2
run 3
average
linux-lts
134.12
154.19
176.85
155.05
linux
87.88
126.88
158.43
124.40
Geekbench6:
I closed all programs except my single firefox tab with google sheets to store the results.
Single-core:
kernel
run 1
run 2
run 3
average
linux-lts
2435.00
2402.00
2418.00
2418.33
linux
2458.00
2344.00
2409.00
2403.67
Multi-core:
kernel
run 1
run 2
run 3
average
linux-lts
16277.00
16426.00
16268.00
16323.67
linux
16238.00
16289.00
16379.00
16302.00
As can be seen above, the results for Dota 2 on Linux 6.10.2 are dramatically worse than on Linux-LTS 6.6.42. Rise of the Tomb Raider also seems to be underperforming, but not nearly to the same extent. Cyberpunk 2077 and Geekbench scores are close enough that I would say the issue does not apply to these tests.
Log files (for system lockups / game freezes / crashes)
I have since done some testing on various Linux kernel versions and have determined that 6.9.10 was the last version that had the ~500fps for dota 2, and starting with 6.10 the performance drops to the 80-100 fps range.
I'm not sure what I am to be expecting, but gimp seems to perform fine for me. I tried creating a new canvas, painting a doodle and saving and all of the ui remained responsive. Maybe its not the same issue?
Grab the PKGBUILD repo for linux: pkgctl repo clone --protocol=https linux (or try getting linux-cachyos: https://aur.archlinux.org/packages/linux-cachyos, though I am not sure if the patch will work on 6.10.3, haven't tested it yet)
cd into the newly created linux directory
Download the patch and put it there
Add the patch to source in the PKGBUILD, like so:
source=( https://cdn.kernel.org/pub/linux/kernel/v${pkgver%%.*}.x/${_srcname}.tar.{xz,sign}$url/releases/download/$_srctag/linux-$_srctag.patch.zst{,.sig} config # the main kernel config file revert-clear-page.patch # this is the important change)
Also comment out the docs builder, since you probably won't need it and it will speed up compilation:
build(){cd$_srcname make all make -C tools/bpf/bpftool vmlinux.h feature-clang-bpf-co-re=1# make htmldocs}
Change the kernel package name in the PKGBUILD to something other than linux not to break compatibility:
pkgbase=linux-something # can be linux-anything
Build and install the new kernel with makepkg -sir (you might also need --skipchecksums --skippgpcheck if the checks fail, I still haven't found a proper way to do it)
With the help of the devs of cachyos I built 6.10.3 with the patch you provided. It has resolved the issue with gimp when moving the window, but Dota 2 is still getting 110-130fps with the same setup specified in the original post. So it Looks to be unrelated.
Alright, thanks for testing! It's a shame it didn't help you
This raises more questions though; I'll need to dig deeper.
As for potential workarounds (apart from staying on 6.9) - one thing I'd like to ask that you didn't mention in the issue description: are the games CPU or GPU bound on 6.10?
If they are CPU-bound, try running the OS with the amd_pstate=active kernel parameter and set "Performance" mode in KDE under Power Management:
(If you don't see Power Management, install cpupower and try sudo cpupower frequency-set -g performance.)
That helps me with some frametime stability issues that I had in some CPU bound games (like BeamNG). Maybe it could help you as well?
If they are GPU bound, you could try using the mesa-tkg-git package from the AUR (currently at 24.3.0-devel) and see if you get some performance improvements, maybe there's some new stuff for 6.10 in newer Mesa commits.
That aside, I'm out of ideas. Hopefully a dev will give us some insight tomorrow
I have a similar performance drop in Dota2 with kernel 6.10.2.
On maximum graphics settings instead of 100-140 fps, which I usually get with kernel 6.8.12, on 6.10.2 the game barely reaches 35-45 fps. my GPU is R9 290X. (amdgpu, nobara 40 gnome)
Yeah, but without it Dota 2 still has performance issues apparently (see above), so there seems to be more to it than just drm/buddy. This particular regression might not be VRAM related.
@Qalli could you send a screenshot of htop with kernel processes shown so that we get a comparison between 6.9 and 6.10?
Uncheck this box:
And click CPU% to sort by CPU usage:
The most interesting parts are:
are there huge red bars (kernel CPU time)?
is Xwayland maxing out one CPU thread ("100%" in htop)?
are the highest CPU time consuming kernel processes (greyed out) TTM-related?
Demo from the video I sent above:
If not, this might be something completely different.
Hmm, I am unable to reproduce this. I retraced your steps, set everything the same as you did. The game is properly GPU bound on my system, similar to Portal 2 (also running 6.10.3 with Alex's patches):
I'll check different Mesa versions to be sure. Done, 24.1.5 is also ok for me.
Do you use CoreCtrl or LACT for amdgpu power profile switching? I do; I always set the 3D_FULL_SCREEN mode. Maybe there's an issue with default power profiles being applied differently between 6.9 and 6.10 somehow? That's probably not it either, since the GPU isn't fully utilized anyway.
(Also, unrelated, but I see that you have above 4G/64-bit decoding and resizable BAR disabled (due to only 256M of CPU-visible VRAM), you could also try enabling that (just above 4G is enough); some games take advantage of it.)
No, not anymore. This is fixed for me in GIMP and other Xwayland/GTK apps.
Edit: as a matter of fact, I do (with lxappearance resizing I do get to 100%, but it's not as bad as without these patches on 6.10; Xwayland is no longer using 100% of one CPU thread).
When you enable resizable BAR in the bios, the bios resizes the BAR before the OS even loads so there is nothing the driver or OS needs to do. The resizeable BAR code in the driver and OS is only used in cases when there is enough MMIO space, but the bios has not already resized the BARs.
Not sure what CSM is doing differently. Maybe the CPU cache settings are set wrong for some parts of the address space in that case? Depends on the bios and what it does with that setting.
@agd5f just to clarify, csm was merely stopping resizeable bar from working. Now that I have a working resizeable bar toggle in the bios it has become apparent that the fps loss is from having resizeable bar disabled on 6.10 whereas having resizeable bar disabled on 6.9 was fine for performance.
I can confirm this behavior, I was seeing issues myself with really bad performance in dota2 on 6.10.3, however it turns out I'd reset my bios and accidentally disabled REBAR in the process. With rebar enabled, the performance issue is gone. So I guess there's some poor interaction with rebar disabled and newer kernels.
With these patches the issue seems resolved to me in regard to xwayland UI elements, such as in GIMP or Prusa Slicer.
Without the patches, after using the computer for 20 minutes or so, any applications that are rendering the UI with xwayland are unusable. As a workaround I can switch to X11 or compile the latest kernel with these patches.
Are these patches going to be pushed to the kernel or is this bug ticket still under investigation?
I was helping bisect it in the forum post mentioned on there as well, but she found what she believes is the commit before me, hopefully that is of some help.
That issue was closed as a duplicate of this one, therefore the commit mentioned there must be relevant, if not then I would reopen the other issue for further investigation, and we would still need to bisect this one.
Same issue here on 6.10.x on OpenSUSE TW. 6600xt. Rebuilding kernel with 'revert clear' patch is the only thing that helped. Otherwise, after 15-20 minutes of GPU usage (or resizing GIMP or HexChat windows a bunch of times) the XWayland CPU usage throttles to 80% and the GPU starts displaying images like it's 1990s AOL downloading a bmp file.
Sorry for taking so long to answer but I had free time and wanted to build this on a couple of different distros to see. I built a fresh pull of 6.10.6-200 on Fedora 40 and was able to verify this patch indeed resolves the issue on my end. I also built it on Arch as well using a 6.10.5 and was able to again have positive results.
Any word on what's happening with this (if anything). Using rolling releases is annoying with bugs like this. I've accidentally marked a kernel I built twice now in Zypper as deinstall because it sees it as "obsoleted".
It looks like occasional frametime spikes became more severe in Cyberpunk 2077 with this patch applied. Is there some way to make this optional / configurable? I tried re-testing a few times, and without it these spikes happen too but they are more mild.
@shmerl you mean the FPS? I don't see any difference between using and not using a patch for your use case. Could you check if you are hitting this amdgpu_ttm_clear_buffer()->amdgpu_ttm_fill_mem() function [clear on allocation] more with the patch than not using the patch? Thanks.
It's dipping of fps (or spiking of frametime - the reverse of it) that occasionally happens that seems more pronounced with this patch.
How exactly can I check it, I need some profiler?
To trigger the case with Cyberpunk 2077, I'm testing a specific scenario. There is a big highway in the game. I'm driving on it (in a huge circle) and since it spans basically the whole city, it periodically loads some stuff I suspect which produces frametime spikes since it loads something into memory (that would be my guess). I can try to record a video of this with mangohud showing spikes in the frametime graph.
To clarify - spikes happen without the patch too, except with it they at least seem to be more severe.
Just want to throw in that for me, frame time spikes are almost cut in half with this patch on 6.10.7. Can't test Cyberpunk, don't own it, but AC Valhalla/Mirage, Spider-Man and Jedi: Fallen Order all improve noticably in FPS and fewer spikes in frame time graph. I am using an older Ryzen 2700x CPU (mitigations off), with a newer (relatively) 6900xt GPU, so old zen/zen+ weirdness maybe.
As a bonus, it completely fixes xwayland apps as reported in #3538 (closed). Which if nothing else I'll apply this patch for myself because orca slicer is nigh unusable without it.
OK, I did some tests adding a printk message. Here is what I used:
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.cindex e785f128411d..69d2e4892872 100644--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c@@ -2284,6 +2284,7 @@ int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo, if (r) goto err;+ printk(KERN_INFO "DEBUGG: calling amdgpu_ttm_fill_mem!\n"); r = amdgpu_ttm_fill_mem(ring, 0, addr, size, resv, &next, true, true); if (r)
Without your fixes patch, riding on that highway in CP2077 amdgpu_ttm_fill_mem was called just 3 times in more than 2 minutes (even 3 or 4 minute probably).
With your fixes amdgpu_ttm_fill_mem was called 131 times in 2 minutes. I hope that helps.
@Traj I'm running a Ryzen 7 1800X with an AMD 7800XT so an even bigger gap, but yes same results here, patch resolves both issues. I'm currently running X11 instead of Wayland as a workaround and that appears to work well for now.
So if anyone doesn't want to go through the process of installing a patched kernel, try running X11.
I see no indication this made it in 6.10.8 however the build of 6.10.8 Fedora is presently using in -testing is leaps and bounds better than the last 6 iterations. Does anyone know why this is? Because whatever Redhat did - it didn't resolve the problem but it's a lot better.
@arunpravin24 I'm sorry, I got it backwards. SDMA is able to be maxed out now, before the patches it was between 5-7% when resizing GIMP's main window was very laggy. My bad.
That doesn't change the fact that the patches considerably help with performance.
without patch - SDMA not maxed out - GIMP lag (here the driver causes the app to wait I think, so that's why SDMA is relatively quiet)
with patch - SDMA maxed out - GIMP performing better
I assume the SDMA usage when resizing windows is due to constant framebuffer recreation. This also happens with native Wayland clients, like the Zed editor.
@Qalli Sorry for the ping, but I was able to reproduce the no-ReBAR issue in Dota 2.
Since this ticket was closed today and that issue's root cause hasn't been found yet, maybe you could open another one just for tracking the ReBAR problem? It seems like two issues were mixed together in this ticket and one of them has been solved, so I'd say it's probably better to open a more specific one.
I'll try bisecting the kernel like Michel suggested soon (sadly I haven't had the time to do that yet) and hopefully we'll be able to find the culprit.
Yea, you can't called it "fixed" because it's still have performance issue without enabled re-bar (at least i'm still have this issue with an old mobo and gcn gpu like hawaii r9 290x on kernel 6.11.3. On the most of old motherboards it's almost not possible to enable resizable bar and without it performance in game still very poor. Last kernel with good performance in Dota2 without re-bar i've tested is 6.8.12-201 (nobara 40). I'm talking about x2 + fps performance difference compare to new kernels and it's not ok. I hope in the near future there's more people who's playing this game on old systems without rebar and old kernels (like lts 6.6/6.8 on popular distros) update to 6.11+ and report this problem.
If I'm not mistaken, this issue was never actually resolved. The initial issue with the Dota 2 performance was separate to the cpu useage spiking issue that got patched in the kernel. The Dota 2 issue could be mitigated by using an older kernel or enabling resizeable bar (correctly, csm disabled etc). But the underlying issue that was introduced in the newer kernels was never rectified