RHEL Intermediate #7: Intro to Containers — Podman/Buildah/Skopeo

This post closes the RHEL Intermediate series. The final topic is containers — specifically Podman, which RHEL 9 has adopted as its standard. The commands look familiar to anyone who has used Docker, but the internals are quite different: there is no daemon, containers run with regular user privileges, and systemd integration is first-class. Walking in without understanding these differences makes it easy to carry over Docker habits and run into trouble. This post maps out those differences from an operational perspective.

The position of this post in the RHEL Intermediate series:

• #1 Intro to SELinux — Enforcing/Permissive, labels, troubleshooting
• #2 LVM — PV/VG/LV, snapshots, expansion
• #3 Advanced storage — Stratis, NFS, Samba
• #4 Networking — NetworkManager (nmcli), bonding, teaming
• #5 Log management — journald, rsyslog, log rotation
• #6 Job scheduling — cron, systemd timer, at
• #7 Intro to containers — Podman/Buildah/Skopeo (differences from Docker) ← this post

Docker itself is covered as an introduction in a separate series — Docker Basics. This post assumes a reader who has used Docker at least once and focuses on how to do the same things with RHEL 9’s standard tools.

Why isn’t RHEL 9 Docker #

Starting with CentOS/RHEL 8, Red Hat removed the Docker package from the default repositories. Podman filled that empty role. The reasons are not simple, but from an operational perspective they come down to one clear line.

Docker is a structure where all container privileges are concentrated in the root daemon, Podman is a structure that runs directly with user privileges without a daemon.

The commands are the same so learning cost is nearly 0, but the fact that rootless is the default makes a big difference from an operational perspective.

Podman install and first container #

$ sudo dnf install -y podman
$ podman --version
podman version 4.9.x

The workflow will look familiar to anyone coming from Docker:

$ podman run --rm -it docker.io/library/alpine sh
/ # apk add curl
/ # exit

# raise nginx
$ podman run -d --name web -p 8080:80 docker.io/library/nginx:1.27
$ curl http://localhost:8080

$ podman ps
$ podman logs web
$ podman stop web && podman rm web

It is hard to tell the difference at first glance. The moment the contrast with Docker becomes clear is realizing that all of this ran without root.

rootless containers — what changes #

User namespace #

Rootless Podman runs on top of user namespaces. Root (uid 0) inside the container is mapped to a regular user’s uid on the host. So while a process inside the container appears privileged, it has only regular-user privileges against host resources.

$ id
uid=1000(curtis) gid=1000(curtis)

$ podman unshare cat /proc/self/uid_map
         0       1000          1
         1     100000      65536

How to read: uid 0 inside container = host uid 1000 (me), uid 1~65536 inside container = host uid 100000~165535 (subuid).

For this mapping to work, per-user uid/gid ranges must be configured in /etc/subuid and /etc/subgid. RHEL 9 sets these up automatically when creating users.

$ cat /etc/subuid
curtis:100000:65536

rootless restrictions #

Reduced privileges do mean there are things you cannot do.

• binding to ports below 1024 ✗ — release 80, 443 with sysctl, or place behind a reverse proxy like Caddy/HAProxy
• NFS mount ✗ — NFS is usually blocked in user namespace
• AF_NETLINK restrictions — tools that go deep into host networking may not work
• some MTU/IP forwarding options restricted

If you hit these limits, you can escape to rootful mode (sudo podman ...). The two use completely separated storage.

# rootless
~/.local/share/containers/storage/

# rootful
/var/lib/containers/storage/

This means the same image can end up downloaded twice on the same machine, so the standard in operations is to commit to one side. Prefer rootless when security is the priority; fall back to rootful when tight host network integration is non-negotiable.

Binding to ports below 1024 #

# permanent application
$ sudo sh -c 'echo "net.ipv4.ip_unprivileged_port_start=80" > /etc/sysctl.d/99-podman-rootless.conf'
$ sudo sysctl --system

# afterwards
$ podman run -d -p 80:80 nginx

Images — registries and OCI #

Podman follows the OCI (Open Container Initiative) standard. Compatible with Docker images. The difference is that registry names are written explicitly.

$ podman pull nginx:1.27
?: Please select an image:
   ▸ registry.access.redhat.com/nginx:1.27
     registry.redhat.io/nginx:1.27
     docker.io/library/nginx:1.27

Docker silently assumes docker.io, but Podman asks you to choose from the candidates listed in /etc/containers/registries.conf. In automation scripts, always write the full path (docker.io/library/nginx:1.27) to avoid ambiguity.

unqualified-search-registries = ["registry.access.redhat.com", "registry.redhat.io", "docker.io"]

Red Hat registries #

Two places RHEL users frequently encounter:

• registry.access.redhat.com — public images that can be pulled without authentication (UBI etc.)
• registry.redhat.io — subscription authentication required. Use after logging in with podman login registry.redhat.io

UBI (Universal Base Image) — a line where Red Hat distributes RHEL base images for free. The standard for running RHEL-compatible containers on top of RHEL in operations.

$ podman pull registry.access.redhat.com/ubi9/ubi:latest
$ podman run --rm -it registry.access.redhat.com/ubi9/ubi cat /etc/redhat-release
Red Hat Enterprise Linux release 9.x ...

Buildah — build even without Dockerfile #

Podman supports Dockerfile builds via podman build, which uses Buildah internally. Using Buildah directly gives you finer-grained control.

Dockerfile build #

FROM registry.access.redhat.com/ubi9/ubi:latest
RUN dnf install -y python3 && dnf clean all
COPY app.py /opt/app.py
CMD ["python3", "/opt/app.py"]

$ podman build -t myapp:1.0 .
$ buildah bud -t myapp:1.0 .  # same meaning

If Containerfile exists it’s used, if not Dockerfile is used. Both are OCI standard format.

Buildah script build #

With a Dockerfile, every instruction creates a new layer, and images grow quickly. Buildah gives you an explicit flow — buildah from → start container → make changes → buildah commit — so you control exactly how many layers end up in the final image.

#!/bin/bash
ctr=$(buildah from registry.access.redhat.com/ubi9/ubi)
buildah run "$ctr" -- dnf install -y python3
buildah run "$ctr" -- dnf clean all
buildah copy "$ctr" app.py /opt/app.py
buildah config --cmd '["python3","/opt/app.py"]' "$ctr"
buildah commit "$ctr" myapp:1.0
buildah rm "$ctr"

Even if you split dnf install and dnf clean all across two separate calls, the resulting image is committed as a single layer. A useful technique for keeping image size down in production.

RHEL patterns frequently seen in Dockerfile #

FROM registry.access.redhat.com/ubi9/go-toolset:latest AS build
WORKDIR /src
COPY . .
RUN go build -o /tmp/app ./cmd/app

FROM registry.access.redhat.com/ubi9/ubi-minimal:latest
COPY --from=build /tmp/app /usr/local/bin/app
USER 1001
CMD ["/usr/local/bin/app"]

ubi9/ubi-minimal is a slim base of about 100MB level. Dropping to a non-root user like USER 1001 is the basis of container security.

Skopeo — tool for moving between registries #

skopeo can move or inspect images between registries without unpacking them. It is essential for backups, mirroring, and air-gapped environments.

$ sudo dnf install -y skopeo

Frequently used flows #

$ skopeo inspect docker://registry.access.redhat.com/ubi9/ubi:latest
{
  "Name": "registry.access.redhat.com/ubi9/ubi",
  "Digest": "sha256:abc...",
  "Architecture": "amd64",
  ...
}

$ skopeo copy \
  docker://docker.io/library/nginx:1.27 \
  docker://registry.example.com/mirror/nginx:1.27

# in an internet-connected place
$ skopeo copy \
  docker://docker.io/library/nginx:1.27 \
  dir:/tmp/nginx-1.27

# bundle as tar and move
$ tar czf nginx-1.27.tar.gz -C /tmp nginx-1.27

# in the isolated place
$ tar xzf nginx-1.27.tar.gz -C /tmp
$ skopeo copy \
  dir:/tmp/nginx-1.27 \
  docker://registry.internal/mirror/nginx:1.27

$ skopeo list-tags docker://registry.access.redhat.com/ubi9/ubi

Think of it as collapsing docker pull + docker save + docker load into a single command. Since no daemon is required, it runs cleanly in CI/CD pipelines.

Podman + systemd — quadlet #

The standard way to run containers in production is to manage them as systemd units. Starting with Podman 4.4 in RHEL 9, quadlet was introduced, letting you define containers using familiar systemd-style unit files.

[Unit]
Description=Nginx web container
After=network-online.target
Wants=network-online.target

[Container]
Image=docker.io/library/nginx:1.27
PublishPort=8080:80
Volume=/srv/web:/usr/share/nginx/html:ro,Z
AutoUpdate=registry

[Service]
Restart=always

[Install]
WantedBy=multi-user.target default.target

Register with systemd and start:

$ sudo systemctl daemon-reload
$ sudo systemctl start web.service
$ sudo systemctl status web.service
$ journalctl -u web.service -f

systemd automatically converts .container files to .service units at boot via quadlet. The result is a setup where systemd is the first-class manager of the container lifecycle.

It’s possible at the user level the same way — place at ~/.config/containers/systemd/web.container and manage with systemctl --user.

Volume’s :Z — SELinux label #

In the quadlet example’s Volume=/srv/web:/usr/share/nginx/html:ro,Z, Z automatically attaches a container-dedicated SELinux label to the host directory.

The label concepts from #1 Intro to SELinux apply directly to container volumes. Forgetting this option and ending up with a container that cannot read its volume is the most common SELinux accident in container work.

Podman compose — docker-compose compatible #

To use an existing docker-compose.yml without changes:

$ sudo dnf install -y podman-compose
$ podman-compose -f docker-compose.yml up -d

That said, the recommended path in production is to migrate to quadlet — you get systemd integration and boot auto-start in one go.

Auto Update — image auto refresh #

If you write AutoUpdate=registry in quadlet or systemd unit, podman-auto-update.timer (one of the RHEL 9 default timers) checks for new images once a day and automatically pulls and restarts containers.

$ systemctl list-timers podman-auto-update.timer
$ podman auto-update --dry-run

Operational recommendation:

• dev/staging: auto-refresh with AutoUpdate=registry
• production: turn off auto-refresh and use explicit deployment procedures (block unintended minor version changes)

Debugging — when containers don’t come up #

# 1. is the image actually downloaded
$ podman images
$ podman pull <image>

# 2. exact error when attempting container
$ podman run --rm -it <image> sh
# (often raised in background -d, then ends with only "Exited" seen)

# 3. logs of a live container
$ podman logs <name>
$ podman logs --tail 100 -f <name>

# 4. SELinux denial — when :Z was missed on Volume
$ sudo ausearch -m AVC -ts recent
$ sudo journalctl -t setroubleshoot --since "10 min ago"

# 5. port collision
$ sudo ss -tlnp | grep :8080

# 6. quadlet — check conversion result
$ /usr/libexec/podman/quadlet -dryrun

If a container launched with -d exits immediately, drop the --rm flag and inspect podman logs <name>, or run podman run --rm -it <image> sh interactively — that is the fastest way to diagnose what went wrong.

Common traps #

• Docker-style habits — missing the docker.io/library/ prefix asks where to pull from among unqualified-search candidates or pulls a different image. Always use the full path in automation.
• trying port 80 in rootless — bind: permission denied. Release with sysctl or behind reverse proxy.
• missing Volume :Z — container can’t read host directory in SELinux Enforcing environment (RHEL 9 default).
• rootless ↔ rootful storage separation — images pulled by sudo podman pull aren’t visible from regular podman commands. Unify to one side.
• container root ≠ host root — rootless is mapped via user namespace. Don’t forget that the security assumption differs.
• image cache accumulation — run podman system prune -a -f periodically. If you rely solely on quadlet’s AutoUpdate=registry, old images keep piling up silently.

Commands to remember #

Wrapping up #

• Podman = Docker command-compatible + no daemon + rootless default. RHEL 9’s container standard.
• Buildah — image build tool. podman build uses it internally. Supports both Dockerfile/Containerfile + script build.
• Skopeo — tool for moving between registries. Essential for mirroring, air-gap, CI/CD pipelines.
• quadlet — first-class integration of Podman + systemd. Systemd auto-converts .container files to services.
• Operational core traps: full registry path, Volume’s :Z, rootless ↔ rootful separation, the difference between container root and host root.

Wrapping up the series #

This closes the 7 episodes of the RHEL Intermediate series. From SELinux, LVM, storage, networking, logs, scheduling, to containers, we made a round of seven areas frequently encountered in daily operations.

Next comes the RHEL Advanced series, which moves beyond a single machine into the territory of operating multiple machines together — clustering (Pacemaker), high availability, tuning, security hardening, and Ansible automation.

Thank you for reading this far.