RHEL in Practice #6 Wrapping Up the Track: A Reference Architecture

Infrastructure Linux RHEL Red Hat
8 min read

Following the RHEL in Practice track, we learned web servers, databases, containers, monitoring, and automation one piece at a time. Each post covered a single topic from start to finish, but in real operations these pieces do not run in isolation. They mesh together on top of a single server. In this final post of the track, we tie every piece we have covered into one reference architecture, drawing the full picture of how to assemble and operate a small web service on RHEL.

A reference architecture is not some grand diagram. It is a one-page agreement on which parts you stack in which order, what you pin permanently, where you lock things down, and what you back up. Treat this post as both a summary of the whole track and a checklist you can pull out and use directly on the job.

A Small Web Service on a Single Box #

Gather the parts covered in this track and you have a complete small web service on one box. The makeup is as follows.

  • nginx front. The entrance that receives external requests. It either serves static content directly or acts as a reverse proxy, forwarding requests to the application behind it (#1).
  • PostgreSQL data layer. The database that holds the service’s state. Install it from the AppStream module and align the data directory and SELinux context (#2).
  • Podman container. The application itself runs in a container. Handle it like a systemd service with quadlet so it comes up automatically at boot (#3).
  • Cockpit/PCP monitoring. View server status through the web console and collect performance metrics with PCP (#4).
  • Ansible automation. Instead of doing all of the above by hand once each, codify it in a playbook so you can reproduce the same server identically at any time (#5).

Here is how these five parts layer up inside a single server.

                    [ External users ]
                          │
                    HTTPS (443)
                          │
   ┌──────────────────────┴───────────────────────┐
   │                  RHEL server                   │
   │                                                │
   │   firewalld (only 80,443,22 open)               │
   │   SELinux enforcing (forced at every layer)     │
   │  ──────────────────────────────────────────    │
   │                                                │
   │   [ nginx front ]   ── reverse proxy ──┐        │
   │                                        │        │
   │                              [ Podman container ]│
   │                                  (the app)       │
   │                                        │        │
   │                              [ PostgreSQL ]     │
   │                                  (data)         │
   │                                                │
   │  ──────────────────────────────────────────    │
   │   [ Cockpit,PCP ] status & performance         │
   │   [ Ansible ]     reproduce the whole setup     │
   │                                                │
   │   journald persistent logs │ chronyd time sync  │
   │   3-2-1 backup (DB dump + config + data volume) │
   └────────────────────────────────────────────────┘

Requests flow from top to bottom, firewalld and SELinux wrap that entire flow, and monitoring and automation prop it up from the side. It is a single-box setup, but each layer’s responsibility is clearly divided.

Operations Checklist #

The heart of a reference architecture is not the diagram but operational discipline. We have gathered in one place the principles emphasized over and over throughout the track. Every time you bring up a new server, run down this list from top to bottom.

1. Pin every setting permanently #

A setting applied only at runtime vanishes with a single reboot. Put mounts in /etc/fstab, services under systemctl enable, the firewall behind --permanent, and SELinux booleans/ports/contexts behind semanage and setsebool -P.

# register the mount in fstab and verify
sudo systemctl daemon-reload
sudo mount -a

# pin services to start automatically at boot with enable
sudo systemctl enable --now nginx postgresql

# check which services come up at boot
systemctl list-unit-files --state=enabled

The criterion is not “does it work now” but “does it still work after a reboot.” The most common operational incident is a temporary setting evaporating along with a reboot.

2. Keep SELinux in enforcing #

When something is blocked, do not flee to setenforce 0 or SELINUX=disabled. Keep enforcing on and solve it with policy. Handle ports with semanage port, file contexts with semanage fcontext + restorecon, and allowed behavior with setsebool -P.

# check enforcing state (must read Enforcing)
getenforce

# diagnose the cause when blocked
sudo ausearch -m AVC -ts recent

Use setenforce 0 only briefly as a check to see whether SELinux is the culprit, and switch back to setenforce 1 immediately.

3. Open firewalld minimally #

Permanently open only the ports you must, and leave the rest closed. For a small web service, the three ports 22 (SSH), 80 (HTTP), and 443 (HTTPS) are usually enough.

# permanently allow only the services you need
sudo firewall-cmd --add-service=http --permanent
sudo firewall-cmd --add-service=https --permanent
sudo firewall-cmd --reload

# check what is open
sudo firewall-cmd --list-all

“Open everything now, lock it down later” almost always stays open forever. Start minimal from the very beginning.

4. Follow the 3-2-1 rule for backups #

Keep 3 copies of the data, on 2 different media, with 1 of them in a different location. For a small web service, the backup targets are three things: the PostgreSQL data (pg_dump), the configuration files (/etc and container definitions), and the container data volume.

# logical DB backup
sudo -u postgres pg_dump appdb > /backup/appdb-$(date +%F).sql

# replicate config and volumes to separate media and an off-site location

A backup is only a backup once you have tested restoring it. A backup you have never periodically verified as restorable is not a backup.

5. Apply updates regularly with dnf #

Do not defer security patches. Refresh regularly with dnf upgrade, and if needed, automate security updates with dnf-automatic.

# preview only the security updates
sudo dnf updateinfo list security

# full refresh
sudo dnf upgrade -y

6. Keep logs with persistent journald storage #

The default journald can lose logs on reboot. Create the /var/log/journal directory to switch to persistent storage, and you can trace logs from before the most recent boot as well.

# create the persistent storage directory, then restart
sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald

7. Operate users with least privilege #

Do not log in directly as root. Grant a regular user only as much sudo as needed, and allow only key authentication for SSH to block password login.

# after registering an SSH key, block password and root login
# in /etc/ssh/sshd_config
#   PasswordAuthentication no
#   PermitRootLogin no
sudo systemctl reload sshd

Start permissions at the smallest amount you can give and expand them when needed. Grant broadly from the start and it is hard to narrow them again.

8. Sync time with chronyd #

Log timestamps, certificate validity periods, and database timestamps all depend on the clock. Sync time with chronyd and check the state.

# check sync state
chronyc tracking
chronyc sources

Security Fundamentals #

Of the checklist above, the four items that pertain to security deserve to be grouped and emphasized separately. Even a small service must not compromise on these four.

  • Keep SELinux enforcing. The last line of defense that stops a compromised application’s impact from crossing the policy boundary.
  • Minimal firewalld openings. Physically reduce the attack surface. A closed port cannot be breached.
  • SSH key authentication. Cut off password brute-forcing at the root. Keep only key authentication and block passwords and root login.
  • Least privilege. Grant each user and service only as much privilege as needed. Even if an incident occurs, you can keep the blast radius narrow.

These four are also the areas RHEL strongly pushes by default. Simply leaving the defaults on instead of turning them off, and operating with them intact, already secures a substantial level of protection.

Learning Path: From Practice to Certification #

With this track, you have built one cycle of RHEL operations into your hands. The path forward splits two ways: going deeper into practice, or organizing your knowledge through certification.

To summarize the flow so far: in the RHEL Essentials track you learned the basic features of systemd, SELinux, firewalld, and storage, and in this RHEL in Practice track you wove those features together and followed scenarios for operating real services. Clear these two stages and the practical instinct needed for certification prep is already largely in place.

A certification is the means to organize that instinct into a system and prove it objectively. Red Hat has two flagship certifications.

  • RHCSA. The system administrator certification. The service management, storage, SELinux, users/permissions, and networking basics covered in this track overlap directly with the exam scope.
  • RHCE. The automation engineer certification. Ansible automation, covered in #5, is the core topic. The order is to earn the RHCSA first, then proceed.

Both exams are hands-on practical exams on real systems, not multiple choice. Hands-on experience building things yourself, as in this track, is the surest preparation.

Closing the Track #

You have completed all six posts of the RHEL in Practice track. Congratulations. You have passed through the reality of RHEL operations — which does not end at installing a single package — in one cycle, from web server to automation. By now you should have your own criteria for what to stack in what order when handed a new server, what to pin permanently, and where to lock down.

This post’s operations checklist is a compressed version of the entire track. Pull it out and run through it each time you bring up a new server on the job, checking that nothing is missing.

Next Steps #

Having gone through the essentials and the practice, it is time to organize your knowledge with certifications. The path continues into two certification tracks.

  • The RHCSA track will organize the system administrator certification’s exam scope item by item and cover the flow for hands-on prep.
  • The RHCE track will guide you through preparing the Ansible-automation-centered engineer certification as the stage after RHCSA.
X