Certified Kubernetes Administrator (CKA) #8 Certificate Management: PKI, kubeconfig, Certificate Renewal

If #7 etcd Backup and Recovery covered how to protect cluster state, this time we cover the TLS certificates that underpin every bit of communication in the cluster. In Kubernetes, all communication exchanged between components is encrypted and mutually authenticated with TLS. When kubectl calls the apiserver, when the apiserver calls etcd, and when the kubelet reports to the apiserver, certificates prove identity.

The catch is that these certificates have a validity period. Most certificates kubeadm issues default to one year, and once they expire, the components can no longer trust one another and the whole cluster grinds to a halt. kubectl suddenly stops working, and the control plane fails to come up. In CKA, certificate management is a core part of the Cluster Architecture domain and a frequent trap in Troubleshooting, so in this post we’ll drill the PKI structure and the renewal procedure into our hands.

The cluster is one giant PKI #

A Kubernetes cluster is itself a single PKI (Public Key Infrastructure). Every component proves its identity with a certificate, and they trust one another by commonly trusting the CA (Certificate Authority) that issued and signed those certificates. The CA is the root of trust. When the apiserver judges “is the kubelet that sent this request really a kubelet,” it checks whether that kubelet’s certificate was signed by the cluster CA.

In this structure, certificates serve two kinds of roles at the same time.

• Server certificates. The receiving side, like the apiserver, proves “I really am that server”
• Client certificates. The connecting side, like the kubelet or controller-manager, proves “I am who I claim to be”

The apiserver is the gateway every other component passes through, so it holds both a server certificate and a client certificate. The apiserver is a server to its clients, but a client to etcd.

The PKI directory structure #

The certificates of a kubeadm-installed cluster all gather under /etc/kubernetes/pki/ on the control plane node. Memorize this directory once and you won’t get lost in the exam.

ls -1 /etc/kubernetes/pki/

ca.crt                      # Cluster root CA certificate (the root of trust)
ca.key                      # Root CA private key (used to sign other certificates)
apiserver.crt               # apiserver server certificate
apiserver.key
apiserver-kubelet-client.crt   # Client cert the apiserver uses to call the kubelet
apiserver-kubelet-client.key
front-proxy-ca.crt          # CA for the aggregation layer
front-proxy-client.crt
sa.key                      # Key pair for signing ServiceAccount tokens
sa.pub
etcd/                       # Directory for etcd-specific certificates

Inside etcd/ are etcd’s own CA and certificates, kept separate. etcd holds the most sensitive data in the cluster, so it uses a separate trust chain.

ls -1 /etc/kubernetes/pki/etcd/

ca.crt                      # etcd-specific CA
ca.key
server.crt                  # etcd server certificate
server.key
peer.crt                    # For communication between etcd members
peer.key
apiserver-etcd-client.crt   # Client cert the apiserver uses to connect to etcd

Who trusts whom #

You can read the trust relationships just from the certificate file names. The key is who was signed by which CA, and whether the other side trusts that CA.

The important point here is that etcd uses a separate CA. For the apiserver to attach to etcd, it needs a client certificate signed by the etcd CA. That’s why apiserver-etcd-client.crt is signed not by the cluster CA but by the etcd CA.

Inspecting the certificate contents directly #

Crack open a single certificate with openssl and the relationships above come into focus. In CKA you’ll often use this command to check expiration and identity (Subject).

openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text

There are three places to look in the output.

• Subject. Who this certificate proves (e.g., CN=kube-apiserver)
• Issuer. Who signed it (CN=kubernetes, i.e., the cluster CA)
• Validity. Not Before and Not After. This Not After is the expiration time

In client certificates, the CN and O (Organization) of the Subject matter especially. Kubernetes interprets the CN as the user name and the O as the group. For example, if you connect with a certificate signed as CN=jane, O=dev, the apiserver recognizes that request as coming from user jane in group dev. This mapping becomes the starting point for authorization in #9 RBAC.

kubeconfig: the wallet that carries your certificates #

If certificate files are the cluster’s proof of identity, then kubeconfig is the config file that bundles that proof with which cluster you connect to and under what identity. kubectl, the controller-manager, and the scheduler all attach to the apiserver through a kubeconfig.

A kubeconfig consists of three blocks plus the context that binds them.

• clusters. The apiserver address (server) and the CA that validates that server (certificate-authority-data)
• users. The identity you connect as. A client certificate (client-certificate-data + client-key-data) or a token
• contexts. The combination binding which cluster, as which user, in which namespace you connect

# View the entire current kubeconfig (sensitive data redacted)
k config view

# The currently active context
k config current-context

# List of contexts
k config get-contexts

The control plane’s kubeconfig files #

kubeadm creates the kubeconfigs that the control plane components will use under /etc/kubernetes/. These files also carry certificates inside, so they become renewal targets too.

ls -1 /etc/kubernetes/*.conf

admin.conf                  # For the cluster admin (kubectl). Usually copied to ~/.kube/config
controller-manager.conf     # For kube-controller-manager to attach to the apiserver
scheduler.conf              # For kube-scheduler to attach to the apiserver
kubelet.conf                # For the node's kubelet to attach to the apiserver
super-admin.conf            # Emergency admin kubeconfig (recent kubeadm)

The client certificate inside admin.conf is signed as CN=kubernetes-admin, O=system:masters. The system:masters group is effectively the highest-privilege group that bypasses RBAC, so this file is a master key to the entire cluster. That’s why admin.conf must never be carelessly copied or exposed.

# ~/.kube/config is usually a copy of admin.conf
sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Checking certificate expiration #

In CKA, certificate problems usually appear in the form “kubectl doesn’t work, find the cause and fix it.” The first step is always checking whether they’ve expired. kubeadm provides a dedicated command.

kubeadm certs check-expiration

CERTIFICATE                EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
admin.conf                 Jun 06, 2027 09:00 UTC   364d            no
apiserver                  Jun 06, 2027 09:00 UTC   364d            no
apiserver-etcd-client      Jun 06, 2027 09:00 UTC   364d            no
apiserver-kubelet-client   Jun 06, 2027 09:00 UTC   364d            no
controller-manager.conf    Jun 06, 2027 09:00 UTC   364d            no
scheduler.conf             Jun 06, 2027 09:00 UTC   364d            no
etcd-server                Jun 06, 2027 09:00 UTC   364d            no
...
CERTIFICATE AUTHORITY      EXPIRES                  RESIDUAL TIME
ca                         Jun 04, 2035 09:00 UTC   9y
etcd-ca                    Jun 04, 2035 09:00 UTC   9y

Two things to note here. Individual certificates are usually valid for one year (a short RESIDUAL TIME), while the CA lasts ten years — much longer. In other words, what you renew periodically is the individual certificates, and the CA is barely touched over the cluster’s lifetime.

In situations where you can’t use the kubeadm command (for example, when the apiserver is already dead), check directly with openssl.

# Just the expiration date, quickly
openssl x509 -enddate -noout -in /etc/kubernetes/pki/apiserver.crt
# notAfter=Jun  6 09:00:00 2027 GMT

# Several certificates at once
for f in /etc/kubernetes/pki/*.crt; do
  echo "$f"
  openssl x509 -enddate -noout -in "$f"
done

Renewing certificates #

If expiration is imminent or already past, renew with kubeadm certs renew. The most common form is renewing all certificates at once.

# Renew all kubeadm-managed certificates at once
kubeadm certs renew all

certificate embedded in the kubeconfig file for the admin to use renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
...
Done renewing certificates. You must restart the kube-apiserver,
kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

Don’t miss the last line. Renewing alone isn’t the end. Control plane components read certificates into memory when they start, so even after you swap the on-disk certificates for new ones, they keep holding the old certificates until you restart them.

Restarting the control plane: move the manifests aside for a moment #

The control plane components run as static Pods (we covered how this works in #2). A static Pod works by having the kubelet watch the manifests in /etc/kubernetes/manifests/ — when a file appears it starts the Pod, and when it disappears it kills it. So moving the manifests out for a moment and then putting them back cleanly restarts the components.

# Move the manifests out for a moment → the kubelet brings down the static Pods
mv /etc/kubernetes/manifests /etc/kubernetes/manifests.bak

# Wait a moment until all components come down, then put them back
mv /etc/kubernetes/manifests.bak /etc/kubernetes/manifests

When you put the manifests back, the kubelet starts the static Pods again, and at this point reads the new certificates. Simply restarting the kubelet itself may not bring the static Pods back up, so moving the manifests is the most reliable approach.

After renewal, you also need to re-copy ~/.kube/config with the new admin certificate so that kubectl connects under the new identity.

sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# Check again that the renewal took effect
kubeadm certs check-expiration

The kubelet certificate renews automatically #

Unlike the control plane certificates, the kubelet’s client certificate usually renews automatically. With rotateCertificates: true (the default) set in the kubelet config, the kubelet requests and swaps in a new certificate on its own as expiration approaches. That’s why the kubelet’s client certificate isn’t among the targets of kubeadm certs renew.

# Check whether auto-renewal is on in the kubelet config
grep rotateCertificates /var/lib/kubelet/config.yaml
# rotateCertificates: true

# The current certificate the kubelet holds
ls -l /var/lib/kubelet/pki/

Thanks to this auto-renewal, you don’t need to renew the kubelet certificate one by one even with many nodes. That said, for auto-renewal to work the kubelet must be able to communicate with the apiserver, so if a node has been off for a long time and comes back with its certificate already expired, renewal can be blocked.

Issuing user certificates: CertificateSigningRequest #

To grant a new admin or developer access to the cluster, you need to issue a client certificate for that person. You could sign directly with the CA key, but Kubernetes provides the CertificateSigningRequest (CSR) resource to handle this through the API. The flow is: the user submits a CSR, an admin approves it, and the cluster returns a signed certificate.

# 1) Generate a private key and a CSR (on the user's side)
openssl genrsa -out jane.key 2048
openssl req -new -key jane.key -out jane.csr -subj "/CN=jane/O=dev"

Here CN=jane becomes the user name and O=dev becomes the group. This identity becomes the RBAC subject as-is.

# 2) Submit the CertificateSigningRequest
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: jane-csr
spec:
  request: <the value of jane.csr encoded with base64 -w0>
  signerName: kubernetes.io/kube-apiserver-client
  usages:
    - client auth

Into request you put the result of cat jane.csr | base64 -w0. signerName specifies the intended use of the certificate to be issued; for a regular user client certificate, use kubernetes.io/kube-apiserver-client.

# 3) Check the submitted CSR → approve it
k get csr
k certificate approve jane-csr

# 4) Extract the signed certificate
k get csr jane-csr -o jsonpath='{.status.certificate}' | base64 -d > jane.crt

Right after approval, the CSR’s status changes to Approved,Issued, and .status.certificate holds the signed certificate as base64. Extract it and save it as jane.crt, and jane’s client certificate is complete.

# 5) Build jane's kubeconfig
k config set-credentials jane \
  --client-certificate=jane.crt --client-key=jane.key --embed-certs=true
k config set-context jane-context --cluster=kubernetes --user=jane

This only covers proving who jane is. What jane can actually do is decided only once you bind permissions with a RoleBinding. Authentication and authorization are separate things, and the authorization side — RBAC — is covered in detail in #9. To clean up a mistakenly created CSR, reject it with k certificate deny or delete it with k delete csr <name>.

Common traps #

The spots where you bleed points in certificate work are almost always the same.

• kubectl doesn’t work with an expired certificate. When the apiserver dies or the client certificate expires, kubectl is blocked with Unauthorized or a connection refusal. At that point, the starting line is to check the on-disk certificates directly with openssl and kubeadm certs, not with kubectl.
• Skipping the restart after renewal. If you only run kubeadm certs renew all and don’t restart the control plane, the components keep using the old certificates. You must move the manifests to bring the static Pods back up.
• Skipping the admin.conf copy. If you don’t re-copy ~/.kube/config from the new admin.conf after renewal, kubectl still tries to attach with the old certificate and fails.
• Forgetting to approve the CSR. If you only submit the CSR and skip k certificate approve, no certificate is issued. Always check whether the status is Pending or Approved,Issued.
• Typos in CN/O. The user name and group are embedded in the certificate’s Subject and signed, so they can’t be changed after issuance. When creating the CSR, write the CN and O of -subj exactly.

Exam points #

• Memorize the layout of /etc/kubernetes/pki/. ca.crt/ca.key are the root of trust, and etcd keeps a separate CA under etcd/.
• A kubeconfig is a combination of clusters + users + contexts; check the structure with k config view and the current identity with k config current-context.
• For expiration, kubeadm certs check-expiration is the default, and if the apiserver is dead, look directly with openssl x509 -enddate -noout -in <file>.
• Memorize renewal as three steps: kubeadm certs renew all → move the manifests to restart the control plane → re-copy ~/.kube/config.
• The kubelet certificate auto-renews via rotateCertificates, so it usually needs no separate work.
• For user certificates, submit a CSR, approve it with k certificate approve, then extract .status.certificate and use it. The identity (CN/O) becomes the RBAC subject.

Wrap-up #

What this post locked in:

• The cluster is one giant PKI, and all communication is TLS mutually authenticated with certificates signed by the CA.
• The PKI directory /etc/kubernetes/pki/ gathers the CA and per-component certificates, and etcd uses a separate CA.
• kubeconfig binds identity and connection target with clusters/users/contexts, and the control plane attaches to the apiserver via *.conf files.
• Expiration checks pair kubeadm certs check-expiration with openssl, and renewal pairs kubeadm certs renew all with a control plane restart.
• User certificates are issued by submitting and approving a CSR, and authentication and authorization are separate.

Next: RBAC #

We’ve established how to prove “who you are” with certificates. Next it’s time to decide what that identity “can do” in the cluster.

In #9 RBAC, we’ll build it ourselves — defining permission bundles with Role and ClusterRole, connecting those permissions to users, groups, and ServiceAccounts with RoleBinding and ClusterRoleBinding, and verifying permissions with kubectl auth can-i. It’s the step that makes the jane we created in this post able to actually do work.