Certified Kubernetes Security Specialist (CKS) #14: Image scan — Trivy, Kubesec, KubeLinter

The CKS series is working through the Supply Chain Security domain. In the earlier #13 Minimal images: distroless, scratch, we narrowed the attack surface by trimming what goes into an image to the minimum. This post covers how you find out what vulnerabilities still remain inside that image.

No matter how small you make an image, if its OS packages or language libraries carry known vulnerabilities (CVEs), the image is dangerous. The heart of supply chain security is knowing what’s inside and filtering out known vulnerabilities before deployment. In this post we cover Trivy, which finds an image’s CVEs; Kubesec, which scores a manifest’s security settings; and KubeLinter, which statically analyzes a manifest’s anti-patterns.

Two kinds of scan: image vs. manifest #

We lump everything together as “image scanning,” but the tools CKS covers split into two by what they inspect. Get this distinction straight first, and you won’t be confused about which tool to reach for in which task.

Trivy finds CVEs by matching the versions of the software inside an image against a vulnerability database. Kubesec and KubeLinter, by contrast, look not at the image itself but at the settings in a YAML manifest. For the same nginx image, Trivy sees the CVEs inside it, while Kubesec sees what securityContext you run it under. The two don’t replace each other — they complement each other.

Trivy: the image vulnerability scanner #

Trivy is an open-source scanner built by Aqua Security and the standard tool for handling image vulnerabilities on the CKS exam. It’s fast, simple to install, and easy to read, which fits the exam environment well.

What it scans #

Trivy scans more than one kind of target. The exam mostly features the first one, image scanning, but it’s worth looking over the rest once.

Basic image scan #

The most basic form just passes the image name.

trivy image nginx:1.18

This prints every vulnerability in every package in that image. But when LOW and UNKNOWN show up too, the output gets long enough that you can easily miss what matters. So on the exam you almost always narrow by severity.

trivy image --severity HIGH,CRITICAL nginx:1.18

Filter down to just HIGH and CRITICAL with --severity, and only the vulnerabilities that need immediate action remain. The severity values are five levels: UNKNOWN, LOW, MEDIUM, HIGH, CRITICAL.

Reading the results #

Trivy’s output splits into tables by the targets in the image (e.g., OS packages, language libraries). The key columns in each row are as follows.

The most important thing here is to gauge risk with Severity and judge whether action is possible with Fixed Version. If there’s a Fixed Version, you bump the package or base image to that version or higher; if it’s empty, the vulnerability has no patch yet, so you need a different mitigation.

Output format and counting only #

The default is a human-readable table, but JSON is handier for automation. When the exam asks for a “vulnerability count,” switching the format to count is faster.

# output as JSON
trivy image --format json --output result.json nginx:1.18

# filter to just CRITICAL as a table
trivy image --severity CRITICAL nginx:1.18

Building a CI gate with exit-code #

Trivy’s real value is in stopping a build when vulnerabilities exist in a CI pipeline. Give it --exit-code, and Trivy returns a non-zero exit code when a vulnerability matching the conditions is found.

trivy image --exit-code 1 --severity HIGH,CRITICAL myapp:1.0

This command ends with exit code 1 if there’s even one HIGH or CRITICAL vulnerability. Check this exit code in CI, and you automatically stop a vulnerable image from moving on to the deployment stage. Add --ignore-unfixed alongside it, and vulnerabilities without a patch yet are excluded from the gate, so the build only stops on things you can fix right now.

trivy image --exit-code 1 --ignore-unfixed --severity CRITICAL myapp:1.0

DB updates and offline #

Trivy downloads a vulnerability DB on its first run. The exam environment may restrict network access, so you need the instinct to check whether the DB is already downloaded. With --skip-db-update, you skip the update and scan only with the DB you already have.

Kubesec: manifest security score #

Kubesec is a tool that scores the security settings of a Kubernetes manifest, not an image. Feed it a manifest, and along with a score it tells you the risky items that lowered the score and the recommended items that would raise it.

kubesec scan pod.yaml

The output is JSON, with these as the key parts.

• score. The manifest’s security score. Higher is safer
• scoring.advise. Settings worth adding to raise the score (recommendations)
• scoring.passed. Settings already applied well
• scoring.critical. Risky settings to fix immediately

Most of the criteria Kubesec scores on are hardening items in securityContext. For example, settings like the following raise the score.

In other words, following Kubesec’s recommendations hardens your manifest in the same direction as the securityContext hardening you saw in #9 Pod Security Admission. Where Trivy looks at “what’s dangerous inside the image,” Kubesec looks at “how safely you run this workload.”

KubeLinter: manifest static analysis #

KubeLinter is a tool built by StackRox that statically analyzes manifests and Helm charts to catch security and operational anti-patterns. Where Kubesec scores, KubeLinter — being a linter — shows you rule violations as a list.

kube-linter lint deployment.yaml

The default rules include items like the following.

• Whether the container runs as root
• Whether readOnlyRootFilesystem is unset
• Whether resource requests/limits are missing
• Whether the latest tag is used
• Whether it holds a dangerous capability

KubeLinter fits well for attaching to CI to filter out anti-patterns before a manifest is merged. When there’s a violation, the exit code is non-zero, so you can build a gate the same way as with Trivy’s exit-code.

The role differences among the three tools #

Putting the three tools covered in this post into one table lets you decide, on the exam, which tool to reach for just by looking at the task’s keywords.

The key distinction is Trivy for image CVEs, the other two for manifest settings. When the task wording says “CVE,” “vulnerability,” “severity,” or “image scan,” it’s Trivy; when it says “securityContext,” “security score,” or “manifest inspection,” it’s Kubesec or KubeLinter.

Exam staple tasks #

1) Find an image with a vulnerability of a given severity #

A common type gives you several images and asks you to pick out the one with a CRITICAL vulnerability. Just scan each image narrowed by severity.

trivy image --severity CRITICAL nginx:1.18
trivy image --severity CRITICAL nginx:1.27

The image where CRITICAL appears is the dangerous one. There’s also a variant asking you to pick the image with the fewest vulnerabilities, so you need to be ready to compare result counts.

2) Replace a vulnerable image with a safe version #

Once you’ve found the dangerous image, the next task is to change the Deployment that uses it to a version with no CVEs or fewer of them.

# scan candidate versions to confirm a safe one
trivy image --severity HIGH,CRITICAL nginx:1.27

# replace the manifest's image tag with the safe version, then apply
kubectl set image deployment/web nginx=nginx:1.27

After the swap, it’s good to scan the new image once more to confirm the high-severity vulnerabilities are gone as intended.

3) Check a manifest by score #

There’s also a type that gives you a manifest and asks you to check its security score, or to apply the recommended items to raise the score.

kubesec scan pod.yaml

Adding the items listed under scoring.advise (runAsNonRoot, readOnlyRootFilesystem, etc.) to securityContext raises the score.

Exam points #

• Trivy is the image CVE scanner. trivy image --severity HIGH,CRITICAL <image> is the basic form, and narrowing severity with --severity is used in nearly every exam task.
• Read the results with Severity and Fixed Version. If there’s a Fixed Version, bump to that version or higher to fix it; if it’s empty, the vulnerability has no patch yet.
• Build a CI gate with --exit-code 1. When a vulnerability of the specified severity exists, it ends with a non-zero exit code, blocking deployment of the vulnerable image. With --ignore-unfixed you can exclude vulnerabilities that have no patch.
• Kubesec is the manifest security score. kubesec scan scores the degree of securityContext hardening, and scoring.advise gives recommendations.
• KubeLinter is manifest static analysis. kube-linter lint shows anti-patterns like running as root, an unset read-only filesystem, and the latest tag as a violation list.
• Tell the tools apart by what’s inspected. For CVEs/vulnerabilities it’s Trivy; for securityContext/manifest settings it’s Kubesec or KubeLinter.

Wrap-up #

What this post locked in:

• Image scanning comes in two kinds. Trivy looks at the CVEs in the image contents; Kubesec and KubeLinter look at the manifest settings.
• Trivy scans image, filesystem, repo, and config, narrows severity with --severity, and builds a CI gate with --exit-code.
• Kubesec scores the degree of securityContext hardening in a manifest and gives recommendations.
• KubeLinter statically analyzes manifests and Helm charts to catch anti-patterns as a violation list.
• The exam staple is the task of finding an image with a vulnerability of a given severity and replacing it with a safe version.

In #13 Minimal images we shrank the image, and in this post we found that image’s vulnerabilities. The remaining questions are whether that image is really the one you built, and how you prove what’s inside it.

Next — Image signing #

Even if you’ve scanned an image and filtered out its vulnerabilities, the scan is meaningless if someone swaps the image out in transit. In the next post we cover the mechanisms that prove an image’s origin and integrity.

In #15 Image signing: cosign, SBOM, we’ll build it ourselves and cover how to sign an image with cosign and verify the signature at the admission stage to block deployment of unsigned images, plus how to generate and use an SBOM (Software Bill of Materials), the list of every component inside an image.