Certified Kubernetes Security Specialist (CKS) #3: CIS benchmark (kube-bench), component security, Ingress TLS, binary verification

In the previous post we locked Pod-to-Pod traffic down to default deny with NetworkPolicy. This time we move on to hardening the control plane that holds the cluster up. If the apiserver accepts anonymous requests, or the kubelet leaves an unauthenticated read-only port open, then no matter how well you write your NetworkPolicies, the front door is still wide open. This post covers the other half of the Cluster Setup domain — component security and cluster inspection.

The key tool is kube-bench. It automatically runs the industry-standard checklist known as the CIS Kubernetes benchmark against your cluster and tells you, as PASS/FAIL/WARN, which settings are unsafe. On the exam, a recurring task is to look at a FAIL that kube-bench reported and fix the flag on the corresponding component.

What the CIS Kubernetes benchmark is #

The CIS (Center for Internet Security) benchmark is an item-by-item checklist for configuring a particular piece of software securely. The CIS benchmark for Kubernetes divides the configuration files, flags, and file permissions of the control plane components (apiserver, controller-manager, scheduler, etcd) and the node components (kubelet, kube-proxy) into hundreds of items, and for each item it specifies what value is considered safe.

Each item falls into one of two levels.

• Automated: items a tool can read from the configuration file and judge mechanically.
• Manual: items that require policy or operational context and must be checked by a person.

Checking this list by hand, one item at a time, is impractical, so kube-bench, built by Aqua Security, automates the inspection. kube-bench reads the configuration files and process flags on the node it runs on, compares them against each item in the CIS benchmark, and outputs the result as PASS/FAIL/WARN.

Running kube-bench #

kube-bench is usually run directly on a node as a binary, or inside the cluster as a Job. In the exam environment the binary is typically already installed, so you run it on the control plane node and the worker node with the appropriate target specified for each.

To inspect control plane items on the control plane node, give the master target; to inspect node items such as the kubelet on a worker node, give the node target. You can also specify both at once.

# On the control plane node: inspect apiserver, controller-manager, scheduler, etcd
kube-bench run --targets=master

# On the worker node: inspect kubelet, kube-proxy
kube-bench run --targets=node

# Specify both at once (only the components present on that node are inspected)
kube-bench run --targets=master,node

You can also state a specific benchmark version explicitly. The benchmark matching your cluster version is selected automatically, but when it’s off, you specify it yourself.

# Specify the CIS benchmark version directly
kube-bench run --targets=master --benchmark cis-1.23

When the results are long and you only want to see specific items, filter the output with grep, or specify the item number with --check.

# Pick out only the FAIL items
kube-bench run --targets=master | grep -E '\[FAIL\]'

# Inspect only a specific item number
kube-bench run --targets=master --check 1.2.1

Reading kube-bench results #

kube-bench’s output consists of an item number, a status, and a description. There are three statuses.

• [PASS]: the item is set to a safe value. Nothing to touch.
• [FAIL]: the item is set to an unsafe value or is missing. This is what you must fix.
• [WARN]: an item that’s hard to judge automatically. It means a person should check it directly, and Manual items mostly fall here.

The heart of the output is the remediation section that follows each FAIL item. kube-bench tells you not only what’s wrong but also how to fix it, as a command or a configuration-change instruction. On the exam, you just read this remediation as-is and apply it.

[INFO] 1 Control Plane Security Configuration
[INFO] 1.2 API Server
[FAIL] 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual)
...
== Remediations master ==
1.2.1 Edit the API server pod specification file
/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node
and set the below parameter.
--anonymous-auth=false

== Summary master ==
42 checks PASS
8 checks FAIL
12 checks WARN
0 checks INFO

The order in which to read this output is as follows. First check in the Summary how many FAILs there are, then find the [FAIL] items in the body, look at the remediation corresponding to that item number, and finally open the indicated file and fix the flag.

Securing control plane components #

Control plane components run as static Pods in a kubeadm cluster. So you change their configuration not with ordinary kubectl, but by editing the manifest files on the node directly. The manifest locations are as follows.

When you save a file in this directory, the kubelet detects the change and re-creates the corresponding static Pod automatically. No separate apply is needed. The basics of the certificate structure and kubeconfig assume the PKI structure covered in CKA #8 Certificate management.

Key apiserver flags #

The apiserver is the cluster’s front door, so the most items land on it. The flags that frequently show up as kube-bench FAILs are summarized below.

You add the flag to the command array in the manifest, or fix its value.

# /etc/kubernetes/manifests/kube-apiserver.yaml (excerpt)
apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
    - name: kube-apiserver
      command:
        - kube-apiserver
        - --anonymous-auth=false
        - --authorization-mode=Node,RBAC
        - --profiling=false
        - --audit-log-path=/var/log/kubernetes/audit.log
        # ... keep the existing flags ...

After saving, the apiserver restarts and the API stops responding for a moment. Wait until it responds again, then check.

# Check whether the apiserver static Pod came back up
kubectl -n kube-system get pod -l component=kube-apiserver

# If there's a typo in the manifest edit, the Pod won't come up. Check the container logs with crictl
sudo crictl ps -a | grep apiserver
sudo crictl logs <container-id>

The most common accident when editing the manifest is the apiserver failing to come up at all due to an indentation error or a wrong flag value. When this happens, kubectl itself stops working, so you find the cause by viewing the container logs with crictl as shown above.

Securing the kubelet #

The kubelet is the agent running on each node, and it amounts to the node’s front door. The kubelet, too, must block anonymous requests and the unauthenticated read-only port. These are items that frequently come up in kube-bench’s node target.

The kubelet is usually configured with a config file. On a kubeadm node, /var/lib/kubelet/config.yaml is the default location, and you can also check the actual runtime arguments in /var/lib/kubelet/kubeadm-flags.env.

# /var/lib/kubelet/config.yaml (excerpt)
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
authorization:
  mode: Webhook
readOnlyPort: 0
protectKernelDefaults: true

The kubelet is a systemd service, not a static Pod, so after fixing the config you have to restart it manually for the change to take effect.

# After editing the kubelet config, restart
sudo systemctl restart kubelet

# Check the status after restart
sudo systemctl status kubelet

When a flag and the config file specify the same item at the same time, the command-line flag usually wins. When you’re unsure which one is actually in effect, start by checking whether the same flag is in kubeadm-flags.env.

Ingress TLS #

The Cluster Setup domain also includes encrypting traffic that comes in from outside. Attaching TLS to an Ingress protects the segment between the client and the cluster with HTTPS. The procedure has two steps. You create a Secret holding the certificate, and reference that Secret from the Ingress’s tls section.

First, create a tls-type Secret from the certificate and the key.

# Create a TLS Secret from the certificate (tls.crt) and key (tls.key)
kubectl create secret tls web-tls \
  --cert=tls.crt \
  --key=tls.key \
  -n default

Next, connect this Secret in the Ingress’s tls section. The domain written in hosts must match the CN/SAN of the Secret’s certificate for the browser to trust it.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-ingress
  namespace: default
spec:
  tls:
    - hosts:
        - app.example.com
      secretName: web-tls
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web-svc
                port:
                  number: 80

After applying it, check whether TLS actually attached.

kubectl apply -f web-ingress.yaml

# Check whether the TLS host is registered on the Ingress
kubectl describe ingress web-ingress

# Check the handshake and certificate (against the Ingress controller's address)
curl -vk https://app.example.com/ --resolve app.example.com:443:<ingress-ip>

If the host and the Secret name appear in the TLS section of the describe output, the connection is made. If you wrote the Secret name wrong or the Secret is in a different namespace, TLS won’t apply, so be sure to check that the Ingress and the Secret are in the same namespace.

Binary verification #

The most basic part of supply chain security is verifying that a downloaded binary hasn’t been tampered with. When you pull tools like kubectl, kubeadm, or kube-bench from the internet, you compare the checksum published by the distributor against the one you compute yourself. If the values differ, it was either corrupted in transit or tampered with, so you discard that binary.

Kubernetes binaries come with a .sha256 file provided by the distributor. After downloading, you compute it yourself with sha256sum and compare.

# Download the binary together with the official checksum file
curl -LO "https://dl.k8s.io/release/v1.30.0/bin/linux/amd64/kubectl"
curl -LO "https://dl.k8s.io/release/v1.30.0/bin/linux/amd64/kubectl.sha256"

# Method 1: a person visually compares the computed value with the published value
sha256sum kubectl
cat kubectl.sha256

# Method 2: compare mechanically with --check (it must come out OK)
echo "$(cat kubectl.sha256)  kubectl" | sha256sum --check

If --check outputs kubectl: OK, integrity is confirmed. If FAILED appears, don’t trust the binary — download it again. On the exam this appears in a form like “verify whether this binary’s sha256 checksum matches the given value,” and the task becomes answering whether they match or picking out the file that doesn’t.

Deeper supply chain verification such as image signing (cosign) and SBOM is covered in #15, and this post’s sha256 verification is its starting point.

Exam points #

• Fixing kube-bench FAILs is a staple. Run kube-bench run --targets=master, find the FAIL items, read the remediation below them as-is, and fix the corresponding manifest or config — get this flow into your hands.
• The control plane is static Pods. Edit the manifests in /etc/kubernetes/manifests/ and the kubelet re-creates them automatically. No apply is needed.
• The kubelet is a systemd service. After fixing /var/lib/kubelet/config.yaml, you have to restart it directly with systemctl restart kubelet for it to take effect.
• After editing a manifest, if the apiserver doesn’t come up, kubectl won’t work either. Memorize the procedure of finding the cause with crictl ps -a and crictl logs in advance.
• Safe values to memorize: apiserver --anonymous-auth=false, --authorization-mode=Node,RBAC, --profiling=false. kubelet anonymous.enabled=false, authorization.mode=Webhook, readOnlyPort=0.
• Ingress TLS requires the Secret and the Ingress to be in the same namespace. Create it with kubectl create secret tls and reference it via secretName in the tls section.
• For binary verification, compare mechanically with sha256sum --check. Check that OK comes out.

Wrap-up #

What this post locked in:

• The CIS Kubernetes benchmark is a checklist of safe per-component settings, and kube-bench inspects it automatically and outputs the result as PASS/FAIL/WARN.
• The heart of kube-bench’s results is the remediation below each FAIL. It tells you exactly what to fix, in which file, and how.
• Control plane components are configured through the static Pod manifests in /etc/kubernetes/manifests/, and the kubelet through /var/lib/kubelet/config.yaml.
• Lock down the apiserver’s and kubelet’s anonymous authentication, authorization mode, read-only port, and profiling to safe values.
• For Ingress TLS, create a tls-type Secret and reference it from the Ingress’s tls section.
• Verify a binary’s integrity by comparing it against the published checksum with sha256sum.

Next — RBAC least privilege #

With the two posts in the Cluster Setup domain, we’ve locked down the network and the components. Now we move on to the second domain, Cluster Hardening, and narrow down who can do what inside the cluster.

In #4 RBAC least privilege in depth, we’ll work through, hands-on, how to find an overly broad ClusterRole, how to design namespace-scoped least privilege with Role and RoleBinding, how to verify permissions with kubectl auth can-i, and the “allow this ServiceAccount exactly this action and nothing more” type that frequently shows up on the exam.