RHEL Intermediate #5: Log Management — journald, rsyslog, log rotation

In Basics #4 we covered journald and journalctl. This post goes a step further and looks at log handling from an operational angle: retention policies for controlling disk usage, collecting logs on remote servers with rsyslog, and auto-rotating text logs with logrotate.

The position of this post in the RHEL Intermediate series:

• #1 Intro to SELinux — Enforcing/Permissive, labels, troubleshooting
• #2 LVM — PV/VG/LV, snapshots, expansion
• #3 Advanced storage — Stratis, NFS, Samba
• #4 Networking — NetworkManager (nmcli), bonding, teaming
• #5 Log management — journald, rsyslog, log rotation ← this post
• #6 Job scheduling — cron, systemd timer, at
• #7 Intro to containers — Podman/Buildah/Skopeo (differences from Docker)

The log structure of RHEL 9 #

In RHEL 9 two tools work together. journald is the main, rsyslog is the secondary.

   kernel / services / user processes
                │
                │  syslog() / journald API
                ▼
   ┌─────────────────────────┐
   │      journald           │   ← main (structured binary)
   │   /run/log/journal/     │   /var/log/journal/ (persistent)
   └────────────┬────────────┘
                │ forward (option)
                ▼
   ┌─────────────────────────┐
   │      rsyslog            │   ← old text log compatibility + remote send
   │   /var/log/messages     │
   │   /var/log/secure       │
   │   /var/log/maillog      │
   └────────────┬────────────┘
                │ logrotate (cron)
                ▼
       rotated compressed log files
       /var/log/messages-20260420.gz

The core of the flow:

• All logs the kernel and services send are first received by journald.
• journald stores in its own binary format and simultaneously forwards to rsyslog as well.
• rsyslog creates the old text files (/var/log/messages etc.) and can also send to remote syslog servers.
• All text log files are auto-rotated by logrotate at certain sizes / periods.

Operators need to know both. journalctl is the modern interface, but rsyslog is still essential for legacy tooling, external SIEMs (Security Information and Event Management), and integration with other hosts.

journald — going deep #

The everyday commands — journalctl -u, -b, --since — were covered in Basics #4. This time we look at the retention policy and disk usage control that operate behind them.

Volatile vs persistent retention #

By default, RHEL 9 runs journald in volatile mode (/run/log/journal/), so logs are lost on reboot. On production machines, enabling persistent retention is standard practice.

$ sudo mkdir -p /var/log/journal
$ sudo systemd-tmpfiles --create --prefix /var/log/journal
$ sudo systemctl restart systemd-journald

# verify
$ journalctl --list-boots
-2 ...   2026-04-18 09:01:23   2026-04-19 08:55:11
-1 ...   2026-04-19 09:00:01   2026-04-20 08:55:32
 0 ...   2026-04-20 09:00:01   2026-04-20 14:23:01

If --list-boots shows multiple boots, persistent retention is on.

Disk usage control — journald.conf #

Once persistent retention is on, logs can accumulate indefinitely. Explicitly setting an upper limit is central to keeping things under control in operations.

[Journal]
Storage=persistent

# disk usage upper limit
SystemMaxUse=2G
SystemKeepFree=500M
SystemMaxFileSize=128M
SystemMaxFiles=100

# retention period
MaxRetentionSec=30day

# compression / sealing
Compress=yes
Seal=yes

Options explained:

Whichever of SystemMaxUse and MaxRetentionSec is reached first applies. Setting both is safe.

$ sudo systemctl restart systemd-journald
$ journalctl --disk-usage
Archived and active journals take up 1.2G in the file system.

journald cleanup — immediate reclaim #

When you urgently need disk space:

$ sudo journalctl --vacuum-size=500M     # reduce to 500M or less
$ sudo journalctl --vacuum-time=7d       # delete things older than 7 days
$ sudo journalctl --vacuum-files=10      # reduce to 10 files

--vacuum-size is the most frequently used option. Reach for it in operations whenever you need to free space immediately after a disk-full alert.

Utilizing structured logs #

The real advantage of journald. All logs have key-value metadata.

$ journalctl _COMM=sshd                       # logs of the sshd command
$ journalctl _SYSTEMD_UNIT=nginx.service     # specific unit
$ journalctl _UID=1000                        # specific user
$ journalctl _PID=1234                        # specific PID

# see all metadata
$ journalctl -o verbose -n 5

# only nginx errors, today
$ journalctl -u nginx -p err --since today

# sudo commands run by a specific user
$ journalctl _COMM=sudo _UID=1000

# ssh attempts from a specific IP (search message body)
$ journalctl -u sshd | grep "192.168.64.50"

Changing output format #

$ journalctl -o json -n 10           # one-line JSON
$ journalctl -o json-pretty -n 5     # nicely formatted JSON
$ journalctl -o cat -n 100           # message body only
$ journalctl -o short-iso -n 100     # ISO 8601 time

-o json is useful for automation. Combined with jq it becomes a powerful analysis tool.

$ journalctl -o json -u nginx --since today | \
    jq 'select(.PRIORITY == "3") | .MESSAGE'    # extract only err messages

rsyslog — why the old standard is still alive #

journald is the main but rsyslog is still active. Three reasons:

1. External SIEM (Splunk, Graylog, ELK etc.) integration — receives via standard syslog protocol.
2. Remote log collection — gathers logs of multiple hosts on one server.
3. Old text file compatibility — automation / monitoring that look at /var/log/messages, /var/log/secure.

$ systemctl status rsyslog
● rsyslog.service - System Logging Service
     Active: active (running)
     ...

Major text logs #

Familiar files created by rsyslog:

/var/log/secure is the first place to look during a security investigation. Every SSH login and every sudo invocation is recorded here.

Configuration — /etc/rsyslog.conf #

# default modules
module(load="imuxsock")        # receive local system logs
module(load="imjournal")       # receive from journald

# rules (facility.priority + action)
*.info;mail.none;authpriv.none;cron.none    /var/log/messages
authpriv.*                                   /var/log/secure
mail.*                                       -/var/log/maillog
cron.*                                       /var/log/cron
*.emerg                                      :omusrmsg:*

The left side of the rule is facility.priority (which kind of log), the right side is the action (where to). Facility includes auth/authpriv/cron/daemon/kern/mail/user/local0~7 etc., and priority is debug/info/notice/warning/err/crit/alert/emerg.

Additional configuration — /etc/rsyslog.d/ #

The standard practice is to leave the core file untouched and add new rules in separate drop-in files.

# only the local6 facility our app uses, in a separate file
local6.*    /var/log/myapp.log

$ sudo systemctl restart rsyslog

Remote logs — multiple hosts → central server #

When managing multiple machines, collecting logs in one place is standard practice. rsyslog handles this well.

Central server (receiving side) #

# receive on UDP 514
module(load="imudp")
input(type="imudp" port="514")

# TCP too (stability)
module(load="imtcp")
input(type="imtcp" port="514")

# separate by host directory
$template RemoteHost,"/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteHost
& stop

$ sudo firewall-cmd --permanent --add-port=514/udp
$ sudo firewall-cmd --permanent --add-port=514/tcp
$ sudo firewall-cmd --reload
$ sudo systemctl restart rsyslog

With this, logs sent by other hosts are stored in /var/log/remote/<hostname>/<program>.log.

Client (sending side) #

# forward all logs to 192.168.64.10
*.*    @192.168.64.10:514       # @ is UDP
# *.* @@192.168.64.10:514       # @@ is TCP (more reliable)

$ sudo systemctl restart rsyslog

Encrypt with TLS (operational recommendation) #

Plain syslog travels unencrypted over the network. In production, wrapping it in TLS is standard. The flow of installing the rsyslog-gnutls package, grabbing certificates, and using imrelp / omrelp modules. Detailed setup is covered in the Advanced series.

logrotate — the standard for log rotation #

Text log files left alone grow to GBs. logrotate automatically handles rotation (renaming the current file and opening a new one), compression, and deletion of old files once the configured period or size threshold is reached.

logrotate is not a separate daemon but a tool called by cron or systemd timer.

$ systemctl list-timers | grep logrotate
NEXT  LEFT  LAST  PASSED  UNIT                  ACTIVATES
...   ...   ...   ...     logrotate.timer       logrotate.service

$ systemctl cat logrotate.timer
[Timer]
OnCalendar=daily
AccuracySec=1h
Persistent=true

Auto-runs once a day. Old RHEL ran via /etc/cron.daily/logrotate but RHEL 9 has moved to systemd timer.

Configuration structure #

weekly
rotate 4
create
dateext
include /etc/logrotate.d

# example: /etc/logrotate.d/nginx
/var/log/nginx/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 640 nginx adm
    sharedscripts
    postrotate
        if [ -f /run/nginx.pid ]; then
            kill -USR1 $(cat /run/nginx.pid)
        fi
    endscript
}

Options explained:

Writing directly #

To leave my app’s logs to logrotate:

/var/log/myapp/*.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    create 640 myapp myapp
    sharedscripts
    postrotate
        systemctl reload myapp.service > /dev/null 2>&1 || true
    endscript
}

Test / forced execution #

$ sudo logrotate -d /etc/logrotate.d/myapp     # debug — outputs only how it would rotate
$ sudo logrotate -v /etc/logrotate.d/myapp     # verbose — actual execution + detailed output
$ sudo logrotate -f /etc/logrotate.d/myapp     # force — ignore rotation conditions and force

In operations, always simulate with -d first. If the output matches your intent, leave the config in place and let the timer take care of the rest.

Relationship between journald and logrotate #

journald files in /var/log/journal/ are not touched by logrotate. journald rotates directly with its own policy (SystemMaxUse etc.).

logrotate handles text log files created by rsyslog (/var/log/messages etc.) and log files written by user applications. journald manages its own files; logrotate handles rsyslog’s text files — a clean separation of two distinct areas.

Security / compliance perspective #

Frequently encountered requirements in operations:

Log integrity #

If you turn on journald sealing with Seal=yes, log tampering is detected.

$ sudo journalctl --setup-keys
$ sudo journalctl --verify

Retention period policy #

Varies by industry / regulation. Common standards:

Long-term retention is typically offloaded to a remote server, S3, or tape. Keep only short-term logs on the machine itself.

Preventing infinite log growth #

Set both SystemMaxUse and MaxRetentionSec, and also specify logrotate’s rotate N. Monitoring alerts on /var/log usage are also standard.

AlmaLinux / Rocky differences #

All commands in this post work as is. journald / rsyslog / logrotate are RHEL packages as is.

Frequently encountered traps #

“Turned on journald persistent retention but disk fills up” #

If SystemMaxUse is not set explicitly, journald defaults to using up to 10% of the disk. On large disks, that can be a significant amount. Always set the upper limit explicitly.

“journalctl doesn’t show old boot logs” #

Persistent retention is not enabled. Check whether the /var/log/journal/ directory exists.

“Logs don’t arrive at the rsyslog remote server” #

Look at the firewall (514 UDP/TCP) and SELinux together. If you changed the port for SELinux, check that syslogd_port_t matches and register with semanage port if needed.

“logrotate isn’t running” #

Check that the timer is active with systemctl status logrotate.timer. Or simulate with logrotate -d to see which file isn’t rotating and why.

“After rotation the app keeps writing to the old file instead of the new file” #

The application needs to receive SIGHUP and reopen the log file, but the postrotate block is missing the kill -HUP or systemctl reload call. Always include the reload command in the rotation rule.

Frequently used commands at a glance #

Wrapping up #

The flows organized in this post:

• RHEL 9 logs work together with three tools: journald (main) + rsyslog (secondary) + logrotate (rotation).
• The standard is to explicitly turn on journald persistent retention (/var/log/journal/) and grab the disk upper limit with SystemMaxUse / MaxRetentionSec.
• When the disk is urgent, immediate reclaim with journalctl --vacuum-size=....
• Structured log search is the combination of fields like _COMM= / _SYSTEMD_UNIT= + -o json + jq.
• rsyslog is the standard for old text log compatibility and remote syslog collection. The central server + client flow is daily for operations beyond a single host.
• logrotate runs daily via systemd timer. Place rules in /etc/logrotate.d/<app> and always validate with -d simulation.
• Retention policy varies from 30 days to 7 years by industry / regulation, and the flow of moving outside the machine is general.

Next — job scheduling #

With logs under control, the next question is when to run scheduled tasks. logrotate itself — running daily — is a prime example of a systemd timer.

In #6 Job scheduling — cron, systemd timer, at we organize, with a guide on which tool to use in which situation: traditional cron and user crontab, at for one-time scheduled execution, anacron that compensates for the time the machine was off, and systemd timer the modern replacement of cron.