Certified Kubernetes Security Specialist (CKS) #13 Minimal images: distroless, scratch (Supply Chain)

With #12 Pod-to-Pod mTLS: Cilium we wrapped up the Minimize Microservice Vulnerabilities domain. From this post on, we enter the Supply Chain Security domain, which carries a 20% weight. Supply chain security is the work of controlling where the images that land on your cluster come from and what they contain, and the first step is minimizing the image itself. The less it carries, the fewer places there are to attack.

In this post, with Dockerfile examples, we’ll nail down why a large image is risky, what distroless and scratch strip away, how to choose among them versus alpine, and the standard multistage build pattern that separates build tools from the runtime.

Why image minimization is where supply chain security begins #

Supply chain security is the work of controlling “exactly what code runs in my cluster.” That code starts from the container image, and inside the image lives not only your application but also the OS libraries, package managers, shells, and assorted utilities that the base image drags along. These incidental components are precisely the attack surface.

Minimizing an image means keeping only what’s strictly needed to run the application and stripping away everything else. The more you strip, the more these three things shrink.

• CVE count. Fewer packages mean fewer known vulnerabilities. Scan an ubuntu-based image with Trivy and it’s common to surface dozens of CVEs from OS packages that have nothing to do with your application.
• Post-breach blast radius. Without a shell and a package manager, even if an attacker breaks into the container, it’s hard for them to pull down extra tools or spin up an interactive shell.
• Image size. A small image deploys faster and reduces the chance of tampering in transit.

This is why image minimization comes before #14 Image scan or #15 Image signing. Making the very surface you scan and the very target you sign small is the foundation of supply chain security.

How a large image widens the attack surface #

An image based on a full OS is packed with components the application will never touch. The following are notably risky.

The key point is that none of these tools are needed to run the application. A statically compiled Go binary runs on its own without a shell or a package manager. Yet when the base image drags all of this in, these components sit unused day to day and only become tools for an attacker at the moment of a breach. So “don’t bake in what you don’t use” is the principle of minimization.

distroless, scratch, alpine compared #

There are broadly three base choices for minimizing an image. Let’s lay out what each carries and strips away in a table.

The selection criteria are simple. If your application is a static binary, scratch is the smallest. If it has runtime dependencies like libc, certificates, or time zones, distroless — which carries only those — is the safe default. alpine is easy to work with thanks to its shell and package manager, but for that very reason its attack surface is wider than distroless, and musl libc’s particular compatibility issues can arise, so from a security standpoint we favor distroless.

Google provides distroless images under the gcr.io/distroless/ path as per-language variants. There are static, base, cc, java, nodejs, python3, and others, each shipping with :nonroot and :debug tags as well.

Removing build tools with a multistage build #

To minimize an image, you have to separate the tools needed at build time from what’s needed at runtime. The compiler, build dependencies, and source code are needed only at build time, while runtime needs only the resulting binary. The thing that does this separation inside a single Dockerfile is the multistage build.

The principle works as follows.

1. build stage. Build the source into a binary on a heavy base that carries a compiler (e.g., golang).
2. runtime stage. On a minimal base like distroless or scratch, bring in only the binary the build stage produced via COPY --from.

The final image keeps only the contents of the last stage. The compiler, the source code, and the intermediate artifacts all drop out, leaving only the executable and a minimal runtime.

Go application: build stage → distroless #

Go can produce a static binary, which makes it the language best suited to minimization.

# build stage
FROM golang:1.22 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
# Produce a statically linked binary (CGO disabled)
RUN CGO_ENABLED=0 GOOS=linux go build -o /app/server ./cmd/server

# runtime stage: distroless static, non-root
FROM gcr.io/distroless/static:nonroot
WORKDIR /app
COPY --from=build /app/server /app/server
USER nonroot:nonroot
ENTRYPOINT ["/app/server"]

Two things are key here. First, CGO_ENABLED=0 produces a statically linked binary that eliminates even the libc dependency, which lets us use the smallest distroless/static. Second, none of the compilers or tools that the golang base brings in end up in the final image, because COPY --from=build pulls in just the one binary.

For a static binary, you can even drop down to scratch.

FROM scratch
COPY --from=build /app/server /server
# If you need HTTPS calls, copy the CA certificates yourself
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
ENTRYPOINT ["/server"]

scratch really is empty, so if you need CA certificates or time zone files, you have to copy them yourself from the build stage. Because of this hassle, in practice people more often use distroless/static, which already carries the certificates.

Node.js application: build stage → distroless #

Interpreted languages also use multistage to separate the dependency-install step from the runtime.

# build stage: install dependencies
FROM node:20 AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY . .

# runtime stage: distroless nodejs, non-root
FROM gcr.io/distroless/nodejs20-debian12:nonroot
WORKDIR /app
COPY --from=build /app /app
USER nonroot
CMD ["server.js"]

The distroless/nodejs base carries only the node runtime — no npm, no shell. It brings in only the node_modules installed by npm ci in the build stage along with the application code, so no package manager remains in the final image.

Running as a non-root user #

One thing you must always pair with image minimization is non-root execution. When a container runs as root, there’s far more an attacker can do inside it after a breach, and far more room for that to escalate into node privileges.

distroless, with the :nonroot tag, runs by default as a non-root user with UID 65532. On scratch or a regular base, you create and designate a user yourself.

FROM alpine:3.20
RUN addgroup -S app && adduser -S app -G app
USER app
ENTRYPOINT ["/app/server"]

Alongside the image-level USER setting, at the workload level it’s good to not trust the image’s USER and to pin it down once more with the Pod’s securityContext.

securityContext:
  runAsNonRoot: true
  runAsUser: 65532
  allowPrivilegeEscalation: false

runAsNonRoot: true refuses to start the Pod at all if the image is configured to run as root, so it’s a line of defense that pairs with a minimal image. The securityContext itself follows on from what we covered in #9 Pod Security Admission and CKAD #4.

How to debug a shell-less image #

The biggest practical inconvenience of distroless and scratch is that with no shell, you can’t get in via kubectl exec. But removing the shell is the heart of the security, so putting a shell back in just to debug puts the cart before the horse. There are two standard approaches.

1) ephemeral container #

This injects a temporary container carrying debugging tools into the same Pod, without touching the running container. It shares the process namespace of the target container, so you can look into the file system and processes of the shell-less container.

kubectl debug -it mypod \
  --image=busybox:1.36 \
  --target=app \
  --share-processes

Because this enables debugging without adding a shell to the production image, it’s the standard approach recommended by both CKS and CKA. The detailed behavior of ephemeral containers is the same as what was covered in the troubleshooting installment of the CKA track.

2) The distroless :debug tag #

Google distroless provides a separate :debug tag that includes a BusyBox shell. You build with this tag and use a shell only during the development/debugging stage, while production deployments use the regular, shell-less tag — keeping the two separate.

# Use only in debugging builds
FROM gcr.io/distroless/static:debug

The rule is to never leave :debug in a production image. The moment you leave a shell behind, the benefit of minimization is gone.

An exam staple: fixing a Dockerfile #

A common task in CKS’s Supply Chain Security domain is making a given Dockerfile more secure. A typical prompt looks like this.

The following Dockerfile builds in a single stage, uses a large base image, and runs as root. Convert it to a multistage build, apply a distroless base, and modify it to run as non-root.

The pre-fix Dockerfile usually looks like this.

FROM golang:1.22
WORKDIR /src
COPY . .
RUN go build -o /server ./cmd/server
ENTRYPOINT ["/server"]

In this case, the points the grader looks at are clear.

• Did you split into multistage? Is there a FROM ... AS build and a second FROM?
• Did you swap in a minimal base? Is the final stage distroless or scratch?
• Did you bring in only the binary via COPY --from? Are the source or the compiler absent from the final image?
• Does it run as non-root? Is there a :nonroot tag or a USER designation?

The post-fix version takes the same shape as the Go example we saw earlier. On exam day, you earn full marks rather than partial credit only if you also confirm the build actually succeeds and the container starts up properly. If you drop CGO_ENABLED=0 and put a dynamically linked binary onto scratch, it fails to start because libc is missing — so matching the base image to the build options is the trap to watch for.

Exam points #

• Image minimization = shrinking the attack surface. Stripping away the shell, the package manager, and needless OS libraries shrinks both the CVE count and the post-breach blast radius together.
• scratch is an empty base, distroless is only the minimal runtime pieces. A static binary goes on scratch; if you need libc or certificates, distroless is the default. alpine is convenient with its shell and apk, but its attack surface is wider.
• The multistage build is the core technique. Compile in the build stage and bring in only the binary into the runtime stage with COPY --from, and the compiler and source drop out of the final image.
• Pairing with non-root. On top of the distroless :nonroot tag or a USER designation, pin down runAsNonRoot: true with the Pod’s securityContext.
• Debugging a shell-less image. Don’t put a shell into the production image; use kubectl debug’s ephemeral container or the distroless :debug tag.
• The exam staple is Dockerfile refactoring. It asks for multistage conversion + a minimal base swap + non-root all at once, and you need to finish with a successful build and a confirmed startup to earn full marks.

Wrap-up #

Supply chain security is the work of controlling the images that land on your cluster, and it begins with minimizing the image to shrink the attack surface itself. The shell and the package manager that a large image drags in are useless for running the application yet become tools for an attacker at the moment of a breach. distroless and scratch are minimal bases that strip away these incidental pieces, and with a multistage build that separates build tools from the runtime, you can produce a small image that holds only the result, with no compiler and no source. Pair this with non-root execution and ephemeral container debugging, and you get an image that’s small yet operable.

Next: Image scan #

Once you’ve made the image small, the next step is to verify that what remains inside it is safe. No matter how much you minimize, a base library or an application dependency can still carry a known vulnerability.

In #14 Image scan: Trivy, Kubesec, KubeLinter, we’ll run things firsthand to nail down how to scan an image’s CVEs with Trivy, how to filter scan results by severity and gate on them, and the pattern for statically checking a manifest’s security settings with Kubesec and KubeLinter.