Certified Kubernetes Security Specialist (CKS) #10 Secrets Management: etcd Encryption, External Secrets

If #9 Pod Security Admission set up policies that reject dangerous Pods at the door, this post is about how you protect the secret data those Pods handle in the first place. Secret security — one pillar of the Minimize Microservice Vulnerabilities domain — starts from the uncomfortable fact that a Kubernetes Secret is not safe in its default state. In this post we’ll cover etcd encryption at rest, re-encrypting existing Secrets, checking for plaintext, and integrating External Secrets and KMS.

A Secret is not encrypted by default #

There’s a misconception to correct right away. A Kubernetes Secret is not encrypted storage. The data in a Secret object simply goes into etcd base64-encoded, and base64 is a trivial encoding that anyone can reverse with a one-line command.

Let’s confirm the difference firsthand. Create a Secret and decode its value as-is, and the plaintext comes right back out.

# Create a Secret
k create secret generic db-cred \
  --from-literal=password=SuperSecret123

# Check the stored value (base64)
k get secret db-cred -o jsonpath='{.data.password}'
# U3VwZXJTZWNyZXQxMjM=

# base64 decode: the plaintext is exposed as-is
k get secret db-cred -o jsonpath='{.data.password}' | base64 -d
# SuperSecret123

base64 is not a security measure but an encoding for binary-safe transport. In other words, anyone who can reach the etcd data can read Secrets in plaintext. Since etcd is stored on disk, the same goes for an attacker who steals an etcd file or gets hold of an etcd backup.

So Secret security is approached from two directions. First, encryption at rest turns the data going into etcd into ciphertext. Second, access control uses RBAC to minimize who can read a Secret. This post focuses on the first, and wraps up the second together with #4 RBAC least privilege.

The structure of etcd encryption at rest #

Kubernetes provides an encryption at rest feature where the apiserver encrypts data just before writing it to etcd and decrypts it on read. The key is handing the apiserver a configuration file called the EncryptionConfiguration.

The flow is simple.

1. Describe in the EncryptionConfiguration which resources (usually secrets) to encrypt with which provider
2. Pass this file’s path to the apiserver via the --encryption-provider-config flag
3. Restart the apiserver
4. Secrets newly written after the configuration go into etcd as ciphertext
5. Secrets already present need to be written once more to get encrypted (re-encryption)

The step people often miss here is #5. The encryption config applies only to writes after it’s set, so unless you re-save existing Secrets, etcd still holds plaintext.

Provider types #

The providers field in an EncryptionConfiguration is an ordered list. Writes use the first provider in the list, and reads scan from the top for a matching provider. The providers worth memorizing for the exam are these.

identity is a special provider meaning “do not encrypt.” If identity comes first in the list, new writes become plaintext; if you put it last, it acts as a fallback for reading existing data that isn’t yet encrypted. This ordering is a frequent exam trap.

Writing the EncryptionConfiguration #

Let’s write an actual file. This is the most typical form, encrypting secrets with aescbc.

# /etc/kubernetes/enc/enc.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded 32-byte key>
      - identity: {}

Let’s go over the reading order again. Under resources you list the resource to encrypt (secrets), and in providers you put the write provider (aescbc) at the very top. The identity: {} at the bottom is a fallback for reading existing Secrets stored as plaintext before the encryption config. Without this fallback, the apiserver can’t read pre-re-encryption existing Secrets and you’ll hit an outage.

You build the key by base64-encoding 32 random bytes.

# 32 random bytes to base64
head -c 32 /dev/urandom | base64
# Fill this output into the secret field in the YAML above

Wiring it to the apiserver #

Since the apiserver is a static Pod, you edit the manifest directly to add the flag and the volume. The kubeadm path is /etc/kubernetes/manifests/kube-apiserver.yaml.

# /etc/kubernetes/manifests/kube-apiserver.yaml (excerpt)
spec:
  containers:
    - command:
        - kube-apiserver
        - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
        # ...existing flags...
      volumeMounts:
        - name: enc
          mountPath: /etc/kubernetes/enc
          readOnly: true
  volumes:
    - name: enc
      hostPath:
        path: /etc/kubernetes/enc
        type: DirectoryOrCreate

You have to wire up three things together. First, specify the config file path with the --encryption-provider-config flag; second, expose the directory holding that file from the node as a volume; third, mount it inside the container with volumeMounts. If you add only the flag and skip the volume, the apiserver can’t find the config file and fails to start. Since it’s a static Pod, saving the manifest makes the kubelet restart the apiserver automatically.

Check that the apiserver came back up.

# Confirm the apiserver Pod restarted
k -n kube-system get pod -l component=kube-apiserver

# If startup fails, trace the cause in the kubelet log
journalctl -u kubelet -f

Re-encrypting existing Secrets #

This is step #5 emphasized earlier. The encryption config applies only to writes after it, so Secrets already in etcd need to be saved once more to turn into ciphertext. The canonical move is a one-liner that reads every namespace’s Secrets and writes them straight back.

# Read Secrets across all namespaces and replace as-is: re-encryption
k get secrets -A -o json | k replace -f -

get ... -o json pulls all current Secrets, and overwriting them as-is with replace makes the apiserver encrypt them with the new provider at write time and store them in etcd. The data content doesn’t change — only the representation stored in etcd changes from plaintext to ciphertext.

For a large cluster, you could pick specific namespaces or narrow the resource types to reduce the load. For the exam, though, the one-liner above is enough.

Checking for plaintext #

To verify the encryption actually took, you read etcd directly. Going through the apiserver returns the decrypted value, which is meaningless; you need to look at the raw bytes stored in etcd. Query the Secret key directly with etcdctl.

# Connect directly to etcd and hexdump the raw Secret value
ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/db-cred | hexdump -C | head

The criterion is simple.

• If a prefix like k8s:enc:aescbc:v1:key1 appears at the start of the output, encryption succeeded. The bytes that follow are ciphertext no human can read
• Conversely, if plaintext like SuperSecret123 appears as-is, it’s unencrypted. You either skipped re-encryption or the provider order is wrong

This prefix check is the basis for the correct answer to an exam task that says “verify that encryption has been applied.”

External Secrets and KMS #

etcd encryption protects data inside the cluster, but there are times you want to manage the lifecycle of keys and secrets themselves outside the cluster. This is where the KMS provider and the External Secrets Operator come in.

KMS provider #

If you use kms as the provider in the EncryptionConfiguration, the encryption keys are managed in an external KMS (e.g., a cloud KMS, HashiCorp Vault’s transit engine) rather than in etcd or on the node disk. The apiserver encrypts a Secret with a data encryption key (DEK) and then wraps that DEK with the KMS’s key encryption key (KEK). The KEK never leaves the KMS, so stealing only etcd or the node isn’t enough to decrypt.

# KMS provider example (excerpt)
resources:
  - resources:
      - secrets
    providers:
      - kms:
          apiVersion: v2
          name: myKmsPlugin
          endpoint: unix:///var/run/kmsplugin/socket.sock
      - identity: {}

The KMS provider works by connecting a separate KMS plugin process on the node over a socket. It’s the recommended approach in production, and the key benefit is that you don’t write the key in plaintext into a config file the way aescbc does.

External Secrets Operator #

There’s also an approach that goes the other direction. The External Secrets Operator (ESO) is a controller that keeps secrets not in Kubernetes but in an external secret store (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, HashiCorp Vault, and so on), syncs those values, and creates them as Secret objects inside the cluster.

The operational flow is as follows.

1. The source of the secret lives only in the external store (the cluster has no original)
2. Define connection info for the external store with a SecretStore or ClusterSecretStore
3. Use an ExternalSecret resource to map “which external key to pull into which Secret”
4. ESO periodically reads the external values and creates/updates the Kubernetes Secret

This makes the external store the single source of truth for secrets, letting you centralize rotation, auditing, and access control in that store. The exam rarely has you install ESO directly, but it’s worth knowing the concept of “managing secrets outside the cluster” and how it differs from KMS. To summarize: KMS encrypts the data going into etcd with an external key, while ESO keeps the secret’s source itself outside and syncs it.

Minimize Secret access with RBAC #

No matter how tightly you apply encryption, it means less if the set of subjects that can read a Secret is broad. etcd encryption at rest blocks the case where the storage medium is stolen, while RBAC blocks the subjects reading the Secret through the normal path. The two lines of defense don’t replace each other — they have to go together.

Apply the principles covered in #4 RBAC least privilege to Secrets directly.

• Grant get/list/watch permission on Secrets only to the ServiceAccounts and users that truly need it
• Don’t leak Secret access through wildcards (resources: ["*"], verbs: ["*"])
• When only a specific Secret is needed, narrow the target to that Secret with resourceNames

# A Role that reads only a specific Secret
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: app
  name: read-db-cred
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["db-cred"]
    verbs: ["get"]

How you hand a Secret to a Pod affects security too. As we saw in #12 ConfigMap and Secret in depth, a volume mount is a safer choice than environment variables. Environment variables are inherited by child processes and easily exposed by certain tools, whereas a volume mount is easier to control with file permissions and has the added benefit of automatically reflecting rotated values.

Exam points #

In the CKS exam, the overwhelmingly common Secret-security task is enabling etcd encryption at rest. Get the following sequence into your hands.

• A Secret is just base64, not encryption. Remember the premise that it’s stored in etcd at the plaintext level
• Write the EncryptionConfiguration. Put resources: [secrets], and in providers put aescbc (or secretbox/kms) at the top and identity at the bottom
• Add all three to the apiserver static Pod together: --encryption-provider-config flag + volume + volumeMount. Avoid the mistake of failing startup by skipping the volume
• Don’t forget re-encrypting existing Secrets. Write them once more with k get secrets -A -o json | k replace -f -
• Verify directly with etcdctl. If the k8s:enc:aescbc:v1: prefix appears in the raw output it succeeded; if plaintext appears it failed
• Be able to answer precisely the meaning of the provider order (first = write, last identity = fallback for reading existing plaintext)
• For concept questions, be ready to explain in one line the difference between the KMS provider and the External Secrets Operator

Wrap-up #

What this post locked in:

• A Secret is stored in etcd as base64 only by default. base64 is just encoding, not encryption, so plaintext is exposed if you reach etcd
• The EncryptionConfiguration + the --encryption-provider-config flag encrypt secrets at rest. Choose among aescbc/secretbox/kms for the provider and put identity at the bottom as a fallback
• The config applies only to writes after it, so you have to re-encrypt existing Secrets with k get secrets -A -o json | k replace -f -
• Verify by reading the raw value with etcdctl and judging by whether the prefix is present
• The KMS provider manages keys externally, while the External Secrets Operator keeps the secret source in an external store and syncs it
• Separately from encryption, you have to minimize Secret access with RBAC so both lines of defense work together

Next — Isolation (gVisor) #

We protected secret data at the etcd level. Now we move on to isolation, which detaches the workload itself from the host kernel.

In #11 Isolation: gVisor, Kata Containers, RuntimeClass, we’ll build firsthand the risk in the structure where a container calls the host kernel directly, the principle by which gVisor intercepts system calls in user space for isolation, the way Kata Containers separates the kernel with a lightweight VM, and the pattern for putting only specific Pods on a sandbox runtime with RuntimeClass.