RHEL Advanced #1: Boot Process — GRUB2, dracut, Recovery Mode

We open the RHEL Advanced series with the boot process. Day to day you just press the power button and the login prompt appears, so the boot stages are nearly invisible. But when boot stalls, when you need to change kernel options, or when you have to rescue a machine whose root password you’ve forgotten, not knowing what happens between “power on” and “login prompt” leaves you with nowhere to start. This post breaks that sequence into stages and shows how to intervene and fix things at each one.

Position of this post in the RHEL Advanced series:

• #1 Boot Process — GRUB2, dracut, Recovery Mode ← this post
• #2 Kernel Tuning — sysctl, tuned, kdump
• #3 Performance Analysis — sar, top/htop, iostat, vmstat, perf
• #4 SELinux Advanced — Writing Policy, audit2allow
• #5 Security Hardening — auditd, OpenSCAP, FIPS
• #6 Subscription / Satellite / Insights
• #7 Cockpit for GUI Management and Web Console

The Full Flow of a RHEL 9 Boot #

When the power comes on, the following five stages run in order.

1. Firmware (UEFI or BIOS)
       │  Runs the bootloader from the ESP (EFI System Partition)
       ▼
2. GRUB2 (bootloader)
       │  Loads the kernel + initramfs from /boot into memory
       ▼
3. Kernel (vmlinuz) + initramfs
       │  Mounts the real root filesystem with essential drivers
       ▼
4. systemd (PID 1)
       │  Walks the unit dependency tree until default.target
       ▼
5. Login prompt (multi-user.target / graphical.target)

Each stage hands off one thing to the next. The firmware just needs to know where the bootloader is. GRUB2’s job ends once it has the kernel and initramfs in memory. The kernel finishes its part by launching systemd as PID 1; from there it is systemd’s territory. Thinking about boot in these discrete stages makes it much easier to diagnose where something has stalled.

Stage 1 — UEFI/BIOS #

Almost every modern RHEL 9 machine boots via UEFI. UEFI firmware finds the EFI System Partition (ESP) on the system disk and directly invokes the bootloader executable inside it.

$ ls /sys/firmware/efi
config_table  efivars  fw_platform_size  fw_vendor  ...

# directory exists → UEFI; absent → legacy BIOS

The ESP is usually mounted at /boot/efi.

$ mount | grep efi
/dev/nvme0n1p1 on /boot/efi type vfat (...)

$ ls /boot/efi/EFI/redhat/
BOOTX64.CSV  fonts  grub.cfg  grubx64.efi  shimx64.efi

shimx64.efi is the file the firmware calls first. In a Secure Boot environment, shim verifies GRUB2’s (grubx64.efi) signature inside the trust chain rooted at Microsoft’s signature, then launches GRUB2. This is what makes RHEL’s Secure Boot flow work cleanly.

Which bootloader the UEFI firmware launches is recorded in NVRAM boot entries. Inspect them with efibootmgr.

$ sudo efibootmgr -v
BootCurrent: 0001
Timeout: 0 seconds
BootOrder: 0001,0002,0000
Boot0001* Red Hat Enterprise Linux  HD(1,GPT,...)/File(\EFI\redhat\shimx64.efi)
Boot0002* UEFI: USB ...

The same command can also reorder, add, or remove entries. Booting a Live USB and creating a boot entry manually with efibootmgr -c is the last resort when GRUB2 itself is broken.

Stage 2 — GRUB2 #

What GRUB2 (GRand Unified Bootloader 2) does, in one line: load the kernel image (vmlinuz) and initramfs from /boot into memory, hand the kernel its arguments, and pass control over.

$ ls /boot
config-5.14.0-...el9.x86_64
initramfs-5.14.0-...el9.x86_64.img
vmlinuz-5.14.0-...el9.x86_64
loader/
grub2/

Do not edit grub.cfg directly #

The file holding GRUB2’s menu and boot options has two locations. On UEFI it is /boot/efi/EFI/redhat/grub.cfg; on legacy BIOS it is /boot/grub2/grub.cfg. Both are auto-generated files and you do not edit them by hand.

# UEFI
$ sudo grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

# legacy BIOS
$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg

grub2-mkconfig reads from two places to build that output:

• /etc/default/grub — global defaults (timeout, default kernel arguments, etc.)
• /etc/grub.d/ — shell scripts that produce menu entries (rarely touched)

Key fields in /etc/default/grub #

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

Remove quiet from GRUB_CMDLINE_LINUX and add something like console=tty0, and boot messages will stream to the screen as they happen. That is the first move when you need to see where boot stalls.

After editing, you must regenerate the config with grub2-mkconfig for changes to take effect.

grubby — add kernel arguments in one line #

If editing GRUB_CMDLINE_LINUX and rerunning grub2-mkconfig feels like too many steps, grubby handles it in one line and applies the change uniformly to every BLS entry.

# add an argument to every kernel
$ sudo grubby --update-kernel=ALL --args="audit=1"

# remove an argument
$ sudo grubby --update-kernel=ALL --remove-args="quiet"

# inspect the currently booted kernel's arguments
$ sudo grubby --info=$(grubby --default-kernel)

This is the recommended approach. Edit /etc/default/grub directly only when you need to change a truly global option; for everyday kernel parameter tweaks, use grubby.

BLS — Boot Loader Specification #

The default menu management style on RHEL 9. Each boot entry lives as one file under /boot/loader/entries/.

$ ls /boot/loader/entries/
6a5f...77a-5.14.0-...el9.x86_64.conf
6a5f...77a-0-rescue.conf

$ cat /boot/loader/entries/6a5f...77a-5.14.0-...el9.x86_64.conf
title Red Hat Enterprise Linux (5.14.0-...el9.x86_64) 9.4 (Plow)
version 5.14.0-...el9.x86_64
linux /vmlinuz-5.14.0-...el9.x86_64
initrd /initramfs-5.14.0-...el9.x86_64.img
options root=/dev/mapper/rhel-root ro resume=... rhgb quiet
grub_users $grub_users
grub_arg --unrestricted
grub_class rhel

Installing a new kernel triggers kernel-install hooks that automatically add a BLS entry. What grubby actually edits is these files.

Stage 3 — Kernel and initramfs (dracut) #

Once GRUB2 has them in memory, two things work in pairs:

• vmlinuz — the actual kernel image
• initramfs — the temporary RAM disk the kernel uses until it can mount the real root filesystem

Why initramfs is needed #

Suppose the real root filesystem sits on LVM, RAID, NFS, or is encrypted with LUKS. If the kernel shipped drivers for every such combination, it would balloon in size and slow down its security patch cadence. So RHEL picks only the drivers needed to mount root on this particular machine and bundles them into a temporary RAM disk. That is initramfs.

The boot flow:

1. Kernel wakes up in memory
2. Unpacks initramfs to RAM and uses it as the temporary root (/)
3. Uses the tools inside it to activate LVM and mount the real root filesystem temporarily
4. pivot_root switches over to the real root
5. Runs systemd (/sbin/init)

If initramfs is broken or missing a needed driver, boot stops at stage 3. Errors like “Cannot find root device” appear right there.

dracut — the tool that builds initramfs #

dracut builds initramfs in a modular way.

# regenerate initramfs for the current kernel (overwrite)
$ sudo dracut --force

# for a specific kernel version
$ sudo dracut --force /boot/initramfs-$(uname -r).img $(uname -r)

# for every installed kernel
$ sudo dracut --force --regenerate-all

When do you need to rebuild?

• After changing the disk/storage configuration that root sits on (e.g., LVM → plain partition)
• After adding a kernel module that is needed from the boot stage
• After enabling or disabling LUKS encryption
• After changing settings under /etc/dracut.conf.d/

Looking inside initramfs #

$ lsinitrd /boot/initramfs-$(uname -r).img | less

You can see exactly which modules are inside and in what order the hook scripts run. It is the first diagnostic tool to reach for when boot stalls in the initramfs stage.

Configuring dracut modules #

Drop snippets into /etc/dracut.conf.d/*.conf.

add_dracutmodules+=" network "
omit_dracutmodules+=" plymouth "

add_dracutmodules forces inclusion, omit_dracutmodules forces exclusion. Note the spaces on both sides of each value — leaving them out causes the value to run into an adjacent module name and produce results you did not intend.

Stage 4 — systemd as PID 1 #

After initramfs pivot_roots to the real root, /sbin/init (a symlink to /usr/lib/systemd/systemd) starts up as PID 1. From here on it is systemd territory, covered in Basics #4.

default.target #

systemd walks unit dependencies until it reaches default.target. On a RHEL server, multi-user.target is usually the default.

$ systemctl get-default
multi-user.target

$ sudo systemctl set-default graphical.target
$ sudo systemctl set-default multi-user.target

Boot time analysis #

Several tools exist to find out where time is being spent during the systemd stage.

$ systemd-analyze
Startup finished in 2.5s (firmware) + 4.1s (loader) + 3.2s (kernel) + 8.7s (initrd) + 12.4s (userspace) = 30.9s
multi-user.target reached after 12.4s in userspace.

$ systemd-analyze blame
4.512s NetworkManager-wait-online.service
2.103s plymouth-quit-wait.service
1.876s dnf-makecache.service
...

$ systemd-analyze critical-chain
multi-user.target @12.4s
└─sshd.service @11.8s +0.6s
  └─network-online.target @11.7s
    └─NetworkManager-wait-online.service @7.2s +4.5s

blame simply lists each service’s startup time in descending order; critical-chain shows the serial dependency path leading to the default target. The candidates worth investigating when boot is slow live on the critical-chain.

Stage 5 — Recovery Mode and Troubleshooting #

If boot stops somewhere, the question is how to get in. Which option you reach for depends on which stage stalled.

rescue.target vs emergency.target #

multi-user.target  ← normal
        │
rescue.target      ← root shell. local filesystems mounted, no network
        │
emergency.target   ← root shell. only root mounted read-only, almost nothing else up

Press e in the GRUB menu to edit a boot entry, then add a systemd.unit argument at the end of the linux ... line.

linux ($root)/vmlinuz-... root=/dev/mapper/rhel-root ro ... systemd.unit=rescue.target

Ctrl+X boots into a root shell. This mode asks for the root password — if you don’t know it, jump to the next procedure.

Recovering the root password — rd.break #

In the GRUB boot entry edit screen, add the following to the linux line.

linux ($root)/vmlinuz-... ... rd.break enforcing=0

Pressing Ctrl+X boots, and a switch_root:/# prompt appears. The real root has been mounted but pivot_root has not happened yet.

switch_root:/# mount -o remount,rw /sysroot
switch_root:/# chroot /sysroot
sh-5.1# passwd
New password: ...
sh-5.1# touch /.autorelabel       # reset SELinux labels on next boot
sh-5.1# exit
switch_root:/# exit               # resume normal boot

Creating /.autorelabel tells systemd to relabel every file’s SELinux context on the next boot. This takes a few minutes (sometimes longer) but prevents the label of /etc/shadow, modified under enforcing=0, from drifting and causing problems later. Skip it, and the next boot may block SSH login and other access.

When boot stops in the initramfs stage #

Messages like “Warning: dracut-initqueue timeout” mean the root filesystem was not found.

Check in this order:

# if an emergency shell appeared
dracut:/# cat /proc/cmdline       # check kernel arguments — is root=... correct?
dracut:/# ls /dev/mapper/         # are LVM devices visible?
dracut:/# vgchange -ay            # manually activate VGs
dracut:/# blkid                   # inspect disk signatures

The most common cause of landing here is rebooting right after writing a wrong UUID or device path into /etc/fstab. The standard recovery procedure is to boot a rescue ISO, chroot in, and fix /etc/fstab.

When boot stops in the systemd stage #

A service fails just before multi-user.target, and boot never finishes.

# boot in emergency mode (GRUB edit → systemd.unit=emergency.target)
# or, after a normal boot, analyze
$ systemctl list-jobs           # in-flight or stuck jobs
$ systemctl status              # full status tree
$ journalctl -b -1              # logs from the previous boot
$ journalctl -b -1 -p err       # only errors from the previous boot

The -1 in journalctl -b -1 means “one boot ago.” The current boot is -b 0 or just -b. This is the option you reach for most often when diagnosing immediately after a forced reboot following a stalled boot.

Common Pitfalls #

• Editing grub.cfg directly — the next grub2-mkconfig run wipes your changes wholesale. Always go through /etc/default/grub or grubby.
• Forgetting grub2-mkconfig after changing kernel arguments — only editing /etc/default/grub does not take effect on the next boot. Also, the output paths for UEFI and BIOS differ — easy to forget.
• Not regenerating dracut — change the root disk layout but leave initramfs alone, and boot stalls. Make dracut --force a habit right after disk work.
• Skipping /.autorelabel after rd.break — change the root password and just reboot, and SELinux labels drift, sometimes blocking SSH login.
• Forgetting enforcing=0 — using rd.break without enforcing=0 can let SELinux deny modifying the password file.
• GRUB_TIMEOUT=0 — with timeout 0, the menu never appears and you lose the chance to enter GRUB edit. For production machines, at least 3–5 seconds is recommended.
• graphical instead of multi-user — install X on a server and boot will wait for the GUI. Pin servers to multi-user.target explicitly.

Commands Worth Remembering #

Wrap-up #

• Boot has 5 stages — UEFI/BIOS → GRUB2 → kernel + initramfs → systemd → default target. Identifying the failing stage makes diagnosis tractable.
• GRUB2 configuration — manage via /etc/default/grub + grubby. grub.cfg is auto-generated; never edit it directly.
• initramfs — temporary RAM disk used before the real root is mounted. After changing disk layout, run dracut --force.
• Recovery modes — rescue.target (local shell, no network), emergency.target (read-only root shell). Enter via GRUB edit.
• Root password recovery — rd.break enforcing=0 for an initramfs shell → chroot /sysroot → passwd → /.autorelabel.

The next post is kernel tuning, which decides how the kernel behaves right after boot. We cover runtime parameters via sysctl, workload-specific optimization with tuned profiles, and dump capture at kernel-panic time with kdump — all in one pass.