Dealing with drm drivers taking a long time to load/probe, enable use-simpledrm by default?

Recently there have been a number of Fedora bug reports (bug 2183743, bug 2276698, bug 2283839, bug 2312355) about the plymouth boot splash not showing properly on PCs using AMD GPUs.

The problem with plymouth and AMD GPUs is that the amdgpu driver is a pretty big driver, which easily takes up to 10 seconds to load on older PCs. The delay caused by this may cause plymouth to timeout while waiting for the GPU to be initialized, causing it to fallback to the 3 dot text-mode boot splash.

I wrote a blogpost about this documenting 2 workarounds (depending on the systems' config): https://hansdegoede.dreamwidth.org/28552.html

I'm filing this plymouth issue to track getting this fixed without requiring users to have to apply these workarounds.

One option, which was also discussed in #110 to resolve that issue is to make the "plymouth,use-simpledrm" kernel commandline option something which is also configurable from the plymouth config file and changing its default to true.

@superm1 added reply to the blogpost. Let me reply to that here:

It seems most likely to me that the reason for the slow down is the decompression and loading of the larger kernel object.

Right, I believe the most time is actually spend on dynamically linking the ko after it has been decompressed / loaded into mem. That approx. 16MB of code is likely calling kernel functions like dev_dbg() / dev_info(), etc. in a lot of places and all those call sites need to be dynamically patched with the function address of those symbols. At least that is what I think is causing the biggest slowdown.

Considering this is all x86_64, have you considered pushing for a change for Fedora to change the kernel config to have the 5 common open source graphics drivers built into the kernel? This will trade off a bit larger kernel image for everyone in exchange for a smaller initramfs.

That is an interesting suggestion as I discussed in this blogpost "Moving GPU drivers out of the initramfs" I have actually been thinking in the other direction. Delaying loading the GPU drivers until after the real rootfs has been mounted and using simpledrm for the splash.

There have been attempts to move the GPU drivers into vmlinuz in the past, but that has lead to some probe ordering issues with i915. I don't remember the details, but there were (are?) some issue where the i915 code relied on builtin drivers to be initialized before it got loaded and making i915 builtin itself broke that assumption. One issue which I'm aware of at least is that some tablets use a external PWM controller for backlight brightness which has their driver build as a module. The pwm-subsystem actually will do an in kernel modprobe call for the driver when i915 calls pwm_get() for these (with the module name coming from a pwm lookup table). Also the pwm lookup tables are registered by some other drivers, so those need to load first...

As you suggested yourself in #110 I think that for now the low hanging fruit fix is to make "plymouth,use-simpledrm" the default.

I think I will take a go at making "plymouth,use-simpledrm" the default for Fedora 42.

That is an interesting suggestion as I discussed in this blogpost "Moving GPU drivers out of the initramfs" I have actually been thinking in the other direction. Delaying loading the GPU drivers until after the real rootfs has been mounted and using simpledrm for the splash.

Either direction should help this issue, but your proposal has the benefit that UKIs don't need GPU firmware or drivers AT ALL, which I think is great for signed initramfs images.

There have been attempts to move the GPU drivers into vmlinuz in the past, but that has lead to some probe ordering issues with i915.

I suspect this is some specific hardware these days because IIUC this is how ChromeOS does its kernel builds.

The pwm-subsystem actually will do an in kernel modprobe call for the driver when i915 calls pwm_get() for these (with the module name coming from a pwm lookup table). Also the pwm lookup tables are registered by some other drivers, so those need to load first...

Do you think it would be feasible to do a call for testing on a binary to find out where the remaining skeletons in the shed are? I think if it could be a longer- term solution to put amdgpu, i915, nouveau, radeon, and xe into the kernel binary this will generally lead to faster boots.

As you suggested yourself in #110 I think that for now the low hanging fruit fix is to make "plymouth,use-simpledrm" the default. I think I will take a go at making "plymouth,use-simpledrm" the default for Fedora 42.

Presumably you mean making the change to conffile upstream for the next release which would be targeted to Fedora 42, right? Sounds good to me.

First of all sorry for being a bit slow with replying...

That is an interesting suggestion as I discussed in this blogpost "Moving GPU drivers out of the initramfs" I have actually been thinking in the other direction. Delaying loading the GPU drivers until after the real rootfs has been mounted and using simpledrm for the splash.

Either direction should help this issue, but your proposal has the benefit that UKIs don't need GPU firmware or drivers AT ALL, which I think is great for signed initramfs images.

I agree that moving the GPU drivers out of the initrd is probably still a good idea. One problem with this approach is that the kms drivers may not be ready yet when the display-manager starts when only loading them after switching to the real rootfs. gdm has a workaround to wait for drm devices to show up in case the kms driver is not ready yet, but sddm does not have and this is causing problems for Ubuntu where the GPU drivers have already been moved out of the initramfs (except when a disk encryption password is needed then Ubuntu still has the GPU drivers in the initrams), see this launchpad bug and this systemd bug.

Moving the loading of the GPU drivers from the initrd to after the real rootfs has been mounted will also require plymouth,use-simpledrm to still timely show the splash and to have a GUI instead of text-mode fallback for entering disk encryption passwords. So IMHO we should to this in steps:

1. Enable plymouth,use-simpledrm by default for Fedora 42.
2. Consider moving the GPU driver loading to after the switch to the real rootfs for Fedora 43, depending on where we are with the issues which Ubuntu is currently seeing.

Presumably you mean making the change to conffile upstream for the next release which would be targeted to Fedora 42, right? Sounds good to me.

Yes that is what I mean.

Considering this is all x86_64, have you considered pushing for a change for Fedora to change the kernel config to have the 5 common open source graphics drivers built into the kernel? This will trade off a bit larger kernel image for everyone in exchange for a smaller initramfs.

That would mean all systems have to pay the price of the kernel being larger and taking longer to load, even those without any AMD GPUs. It might not make any difference for initramfs on such systems.

My suggestion would be splitting up the amdgpu module.

The first step is splitting off everything that isn't needed for deciding whether or not amdgpu should actually initialize into a separate module. That should eliminate most of the module loading and memory usage penalty on systems where amdgpu doesn't actually initialize.

Longer term, it might make sense to split things up even further, e.g. along IP family boundaries. That would reduce loading time and memory usage even on systems where amdgpu actually initializes.

That would mean all systems have to pay the price of the kernel being larger and taking longer to load, even those without any AMD GPUs. It might not make any difference for initramfs on such systems.

My suggestion is specifically in the context of a UKI. If you have to have a signed initramfs that covers all the GPUs anyway, you might as well put all those drivers into the kernel binary of the UKI instead.

My suggestion would be splitting up the amdgpu module.

Right; this has been raised by Hans in the past too.

Is there even a downside to turning on use-simpledrm by default? Plymouth handles the transition between simpledrm to the real driver when still running, better than anything else I tested, including kwin, xserver, weston, wlroots, mir, and mutter.

My vote would be to go ahead, but I don't have a wide variety of hardware to test...

I had another thought, many distributions are compressing kernel modules, and them putting those compressed modules into compressed initrds. That means when booting, they have to be doubly decompressed, right? That has to be impacting the load time as it has to decompress twice I would assume?

Is there even a downside to turning on use-simpledrm by default?

This is explained in the code (in ply-device-manager.c):

/*
 * Simple-framebuffer devices driven by simpledrm lack information
 * like panel-rotation info and physical size, causing the splash
 * to briefly render on its side / without HiDPI scaling, switching
 * to the correct rendering when the native driver loads.
 * To avoid this treat simpledrm devices as fbdev devices and only
 * use them after the timeout.
 */

This was added by 83b38506 to fix #167 (closed).

So because some devices have problems, simpledrm is disabled on all devices, even though most would probably work fine. (I am using plymouth.use-simpledrm on my laptop without any problems.)

I had another thought, many distributions are compressing kernel modules, and them putting those compressed modules into compressed initrds. That means when booting, they have to be doubly decompressed, right?

Initrd generators can deal with this, either by decompressing the modules, or by putting them into uncompressed section of the initrd. For example mkinitcpio supports both (it can be configured by MODULES_DECOMPRESS option).

Hmm, yeah the rotation might be annoying I guess in those cases, but I do wonder if the default should be flipped anyway, since Plymouth can display diagnostic info now with ply-console-viewer.

And folks that have to use simpledrm, might have to wait longer because of the timeout to see the bootsplash

Is there even a downside to turning on use-simpledrm by default?

This is explained in the code (in ply-device-manager.c):

/*
 * Simple-framebuffer devices driven by simpledrm lack information
 * like panel-rotation info and physical size, causing the splash
 * to briefly render on its side / without HiDPI scaling, switching
 * to the correct rendering when the native driver loads.
 * To avoid this treat simpledrm devices as fbdev devices and only
 * use them after the timeout.
 */

This was added by 83b38506 to fix #167 (closed).

Note that that comment is no longer accurate:

1. The rotation issue was fixed 2 years ago in the kernel by this commit.

2. The HiDPI scaling issue was addressed in plymouth commit 01702de7

It would be great if someone can submit a MR to update the comment

So the 2 main disadvantages of enabling use-simpledrm by default are already addressed, but the cosmetic issues 2. - 5. mentioned in my blogpost about moving the drivers out of the initrd will already show up by just enabling use-simpledrm, see the blogpost.

I could update the comment, but if it's decided that use-simpledrm is not the default, it may have to be changed twice?

I could update the comment, but if it's decided that use-simpledrm is not the default, it may have to be changed twice?

True. I plan to make some changes to this code to allow configuring the behavior from the plymouth configfile instead of only looking at the kernel commandline. I'll also take care of updating the comment when I add the configfile support.

mentioned in issue drm/amd#3636

Thank you @jwrdegoede for addressing the issue related to graphical boot on AMD hardware. Running a test on a 2022 Dell Inspiron 14 7425 equipped with a Ryzen 7 5825U APU running Fedora 41 using plymouth.use-simpledrm on cmdline, I hit an issue related to the fourth disadvantage case mentioned on Moving GPU drivers out of the initramfs blog. Adding omit_drivers+=" amdgpu radeon nouveau i915 "improved a bit but no smooth enough when transitioning from full modeset. It looks line using simpledrm is a logical step once some issues are ironed out.

Running a test on a 2022 Dell Inspiron 14 7425 equipped with a Ryzen 7 5825U APU running Fedora 41 using plymouth.use-simpledrm on cmdline, I hit an issue related to the fourth disadvantage case mentioned on Moving GPU drivers out of the initramfs blog.

You mean this disadvantage from the blogpost? : "On systems where a full modeset is done the monitor going briefly black from the modeset will move from being just before plymouth starts to the switch from simpledrm drm to the real driver. "

IOW the spinner (briefly) shows and then the screen briefly goes black from the modeset before the spinner showing again ?

I assume that on normal boots grub is not showing. So without plymouth.use-simpledrm on the commandline you get "Dell vendor logo", "brief black from modeset, "Dell logo with Fedora spinner", right ? IOW in neither scenario you have a flickerfree experience, but with the modeset now happening after showing the spinner, things look slightly worse?

What happens if you also add amdgpu.seamless=1 to your kernelcommandline ? That option enables a so called flickerfree modeset for the LCD panel, which should avoid the screen going black independent of plymouth.use-simpledrm.

You mean this disadvantage from the blogpost? : "On systems where a full modeset is done the monitor going briefly black from the modeset will move from being just before plymouth starts to the switch from simpledrm drm to the real driver. "

Yes.

IOW the spinner (briefly) shows and then the screen briefly goes black from the modeset before the spinner showing again ?

Yes with both Dell logo and the Fedora logo at the bottom.

I assume that on normal boots grub is not showing. So without plymouth.use-simpledrm on the commandline you get "Dell vendor logo", "brief black from modeset, "Dell logo with Fedora spinner", right ? IOW in neither scenario you have a flickerfree experience, but with the modeset now happening after showing the spinner, things look slightly worse?

Since Fedora is the only system of the laptop, grub is not showing as Dell logo displayed first then both spinner and the Fedora logo followed up smoothly. The flicker happened then both "Dell logo", the spinner and Fedora logo dimmed before going to the login screen.

What happens if you also add amdgpu.seamless=1 to your kernelcommandline ? That option enables a so called flickerfree modeset for the LCD panel, which should avoid the screen going black independent of plymouth.use-simpledrm.

Using amdgpu.seamless=1 to the kernel command line, the screen went black for the remaining of time after both Dell logo and the spinner disappeared. I presume it is a bug related to AMD Vega architecture i.e. Ryzen 7 5825U.

On a side note, my convertible laptop uses systemd-boot instead of GRUB2 and UKIfied kernel. I attached extract of 'journalctl -b --no-hostname | grep amdgpu' amdgpu-boot.log where a brief black screen displayed before the backlight sequence. It is minor.

What happens if you also add amdgpu.seamless=1 to your kernelcommandline ? That option enables a so called flickerfree modeset for the LCD panel, which should avoid the screen going black independent of plymouth.use-simpledrm.

FWIW this should be the default behavior in recent kernels. But there is a pretty long list of requirements to be able to enable it on the boot stream that require cooperation with the platform. Unfortunately; no dynamic debug statements are in that call right now to figure out which one fails so you would have to litter it with printk's to figure out which reason fails. Very notably I want to mention that it only works on eDP. So this behavior that is described is going to happen on any desktop form factor GPU.

How to know amdgpu.seamless=1 handles GCN architecture like VEGA8 found in APU on a Convertible like Dell Inspiron 14 7425? The system is running on 6.13.1-200.fc41.x86_64 as tested

So, I decided to test this on both my host system (a Dell G5 5590 from 2019 with Nvidia Optimus) and my VM (Fedora Silverblue 41 libvirt VM with UEFI + Secure Boot).

On both I get a black screen for a large portion of the boot.

I would guess it might be easy to solve this for the VM? After all the only benefit I think I get from having the GPU driver loaded that early is that it starts supporting spice very early in the boot.

So, I decided to test this ...

What do you mean with "this" did you remove the GPU drivers from the initrd; or did you only add plymouth.use-simpledrm to your kernel commandline?

How was the host system boot experience before you made any changes; and which changes did you exactly make ?

Is your host system using uefi to boot, or are you using classic BIOS boot? Also is your host system using a grub which never shows itself ala: https://fedoraproject.org/wiki/Changes/HiddenGrubMenu / https://hansdegoede.dreamwidth.org/19180.html ?

Showing a black screen for a long time suggests that your bootloader causes the screen to go black instead of leaving the vendor logo in place, combined with plymouth not being able to draw to it until the actual kms driver loads. plymouth.use-simpledrm should actually make this better, but only if there is an EFI framebuffer to draw on which requires using UEFI boot and only if your host kernel has the simpledrm driver enabled as builtin driver.

---

As for the vm, the vm very likely is using classic BIOS boot so there adding plymouth.use-simpledrm will not make any difference since there is no firmware framebuffer for simpledrm to use.

What do you mean with "this" did you remove the GPU drivers from the initrd;

Is it possible to configure the initrd on the Fedora Atomic Desktops? And to actually keep the changes on newer commits?

or did you only add plymouth.use-simpledrm to your kernel commandline?

Yes, I did this.

How was the host system boot experience before you made any changes

Dell logo on boot, a few seconds later the Fedora logo and the spinner appears.

Is your host system using uefi to boot, or are you using classic BIOS boot

UEFI with Secure Boot.

Also is your host system using a grub which never shows itself

Yes:

$ sudo cat /boot/grub2/grubenv
# GRUB Environment Block
# WARNING: Do not edit this file by tools other than grub-editenv!!!
menu_auto_hide=1
boot_success=1
boot_indeterminate=0
<snip>

---

As for the vm, the vm very likely is using classic BIOS boot so there adding plymouth.use-simpledrm will not make any difference since there is no firmware framebuffer for simpledrm to use.

Nope:

  <os firmware="efi">
    <type arch="x86_64" machine="pc-q35-8.1">hvm</type>
    <firmware>
      <feature enabled="yes" name="enrolled-keys"/>
      <feature enabled="yes" name="secure-boot"/>
    </firmware>
    <loader readonly="yes" secure="yes" type="pflash">/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd</loader>
    <nvram template="/usr/share/edk2/ovmf/OVMF_VARS.secboot.fd">/var/lib/libvirt/qemu/nvram/silverblue_VARS.fd</nvram>
    <boot dev="hd"/>
  </os>

I should be able to easily record a video of how it currently looks on the VM if you think it's worth it.

What do you mean with "this" did you remove the GPU drivers from the initrd;

Is it possible to configure the initrd on the Fedora Atomic Desktops? And to actually keep the changes on newer commits?

You did not mention you are also using Fedora Atomic for the host. AFAIK on Fedora Atomic the initrd's are still generated at kernel install time and not pre-generated on the buildsystem (for now).

But not making any initrd changes is good, I just wondered if you did make any changes.

or did you only add plymouth.use-simpledrm to your kernel commandline?

Yes, I did this.

Ok.

How was the host system boot experience before you made any changes

Dell logo on boot, a few seconds later the Fedora logo and the spinner appears.

Ok, so the Dell logo stays in place all the time or briefly goes black just before the spinner shows I guess?

And now with just adding plymouth.use-simpledrm as only change you "get a black screen for a large portion of the boot." ?

Hmm, how do you add plymouth.use-simpledrm to your kernel commandline. Have you edited the BLS configs under /boot/loader/entries/*.conf by doing e.g. :

sudo grubby --update-kernel=ALL --args="plymouth.use-simpledrm"

or are you editing the cmdline by using the grub edit cmdline option during boot? If you are editing the cmdline during boot then the use of grub will destroy the vendor logo on the framebuffer and getting a black screen is expected.

If you have not edited the BLS configs already please run:

sudo grubby --update-kernel=ALL --args="plymouth.use-simpledrm"

(I think this will work on an atomic setup too)

and then do a clean reboot from the GNOME system menu.

If you already modified the BLS configs and are doing a clean reboot already then something strange is going on.

In that case can you please file a bug for this against plymouth in Fedora's bugzilla: https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora&component=plymouth

boot with log_buf_len=50M plymouth.debug=stream:/dev/null added to your kernel commandline on top of the plymouth.use-simpledrm and then do a clean reboot. This will create a /var/log/plymouth-debug.log file, please attach this to the Fedora bug.

You did not mention you are also using Fedora Atomic for the host. AFAIK on Fedora Atomic the initrd's are still generated at kernel install time and not pre-generated on the buildsystem (for now).

Yeah, I forgot about this detail. It's Fedora Silverblue 41 both on the VM and on the host.

But not making any initrd changes is good, I just wondered if you did make any changes.

Got it.

Ok, so the Dell logo stays in place all the time or briefly goes black just before the spinner shows I guess?

I will re-test later today and record a video, might be the best solution.

And now with just adding plymouth.use-simpledrm as only change you "get a black screen for a large portion of the boot." ?

Yeah, from what I noticed there's some chunk of time with black screen that didn't happen before. I do not remember right now whether the black screen was before or after LUKS password prompt.

Hmm, how do you add plymouth.use-simpledrm to your kernel commandline. Have you edited the BLS configs under /boot/loader/entries/*.conf by doing e.g. :

sudo grubby --update-kernel=ALL --args="plymouth.use-simpledrm"

or are you editing the cmdline by using the grub edit cmdline option during boot? If you are editing the cmdline during boot then the use of grub will destroy the vendor logo on the framebuffer and getting a black screen is expected.

If you have not edited the BLS configs already please run:

sudo grubby --update-kernel=ALL --args="plymouth.use-simpledrm"

(I think this will work on an atomic setup too)

and then do a clean reboot from the GNOME system menu.

We don't have grubby on Fsdora Atomic Desktops, we have rpm-ostree kargs.

I used rpm-ostree kargs --editor and my current kargs is this:

$ rpm-ostree kargs
rd.luks.uuid=luks-<snip> rhgb quiet root=UUID=<snip> rootflags=subvol=root,compress-force=zstd:6 rw rd.driver.blacklist=nouveau modprobe.blacklist=nouveau nvidia-drm.modeset=1 intel_iommu=on nosgx mem_sleep_default=deep plymouth.use-simpledrm ostree=<snip>

---

I will attach a video of the host and VM boot later today.

I will attach a video of the host and VM boot later today.

This is getting a bit off-topic for this issue. Please file a Fedora bugzilla, with the requested plymouth-debug.log as described in my previous reply.

@jwrdegoede Filled https://bugzilla.redhat.com/show_bug.cgi?id=2339068 for the VM and https://bugzilla.redhat.com/show_bug.cgi?id=2339072 for the host system.

Hello. I am also experiencing this issue. Setting plymouth.use-simpledrm in GRUB for kernel command line does make plymouth show instantly after grub exits, but it still blackscreens for a second after that. Without plymouth.use-simpledrm there are a couple of seconds delay between when grub exits and plymouth splash is shown. amdgpu.seamless=1 doesn't make it better with or without plymouth.use-simpledrm.

My system:
Lenovo Legion 5 (15ACH6H)
OS: Arch Linux
Kernel Version: 6.13.2-zen1-1-zen (64-bit)
plymouth 24.004.60-10

mentioned in merge request !342