Certified Kubernetes Administrator (CKA) #24 Troubleshooting 3: Control Plane (apiserver/etcd/scheduler Down), etcd Recovery

The previous two troubleshooting posts rested on the premise that the cluster was alive. In both the Pod failures of #22 and the node failures of #23, kubectl responded, so we could read symptoms with describe and logs. This post sets a different stage. When the control plane itself collapses, kubectl stops working entirely — and you have to narrow down the cause with your primary diagnostic handle gone.

Fortunately, the control plane has a consistent structure. In a cluster built with kubeadm, apiserver, etcd, scheduler, and controller-manager all run as static Pods. This single fact is the key that runs through all of control plane troubleshooting. Once you understand how static Pods work, it becomes clear what to look at even when apiserver is dead and kubectl is frozen. In this post, we’ll organize those diagnostic tools, the symptom-by-symptom flow, and how to bring a broken control plane back to life.

The starting point of diagnosis: the control plane is static Pods #

To solve control plane troubleshooting, you have to pull out one fact we covered in #2 again. In a kubeadm cluster, control plane components come up not as a Deployment or DaemonSet, but as static Pods. A static Pod is a Pod that comes up without going through apiserver — each node’s kubelet watches a manifest directory on disk directly and brings it up. The manifests live here.

ls /etc/kubernetes/manifests/
# etcd.yaml
# kube-apiserver.yaml
# kube-controller-manager.yaml
# kube-scheduler.yaml

Two conclusions follow from this structure. First, since kubelet reads these files directly, fixing a file makes kubelet recreate the Pod automatically. No kubectl apply needed. Second, since static Pods don’t go through apiserver, kubelet keeps watching the manifests even if apiserver dies. That is, even when apiserver is entirely down and kubectl is frozen, going into the node and fixing the manifest makes kubelet bring the component back up.

That’s why control plane troubleshooting almost always comes down to the same place: SSH into the control plane node and inspect the manifest directory, the kubelet logs, and the container runtime directly.

Diagnostic tools when kubectl is frozen #

When apiserver dies, neither kubectl get nor kubectl describe responds. A message like this is typical.

The connection to the server 10.0.0.10:6443 was refused - did you specify the right host or port?

At this point you drop kubectl and go down to the tools on the node. Three of them are essential.

crictl: look at the containers directly #

crictl is a command that talks directly to the CRI runtime kubelet uses. Since it doesn’t go through apiserver, it works even when apiserver is dead. This is where you check whether the control plane containers are up, or dying and restarting in a loop.

# All container states (including stopped ones)
crictl ps -a

# Control plane containers only
crictl ps -a | grep -E 'apiserver|etcd|scheduler|controller'

If the apiserver container doesn’t show up in crictl ps, or it’s in Exited state and restarting in a loop, that means apiserver is trying to come up and failing. Read the logs directly by container ID.

# Logs of the dead container (the failure cause is printed here)
crictl logs <container-id>

journalctl: see through kubelet’s eyes #

The thing that brings up static Pods is kubelet. If a manifest has an error, kubelet records that fact in its logs.

journalctl -u kubelet -f

Problems like a YAML syntax error in the manifest, a wrong flag, or a path to a nonexistent file show up directly in the kubelet logs.

the manifest file: 80% of causes are here #

Most problems where the control plane won’t come up originate from a defect in the manifest itself. Suspect three things.

• YAML syntax typos. Misaligned indentation, a missing colon, mismatched quotes. If kubelet can’t even parse the file, the Pod won’t come up at all.
• A wrong flag or value. A misspelled flag, a wrong port, a path to a nonexistent certificate. The container comes up and dies immediately.
• A wrong image tag. Write a nonexistent version and it gets blocked on ImagePull.

It’s a safe habit to back up the original before editing.

cp /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml.bak

Symptom 1: apiserver down (kubectl unresponsive) #

This is the most urgent symptom. kubectl stops entirely, so you can’t operate the cluster. There are roughly three causes for apiserver not coming up.

Cause A: manifest error #

A YAML typo or wrong flag in /etc/kubernetes/manifests/kube-apiserver.yaml is the most common. For example, writing the --etcd-servers value wrong, a typo in a certificate path, or misaligned indentation.

# If the container keeps coming up and dying, look at the dead container's logs
crictl ps -a | grep apiserver
crictl logs <apiserver-container-id>

The logs print which flag or path is the problem. Fix the manifest and save, and kubelet recreates apiserver with the new arguments. If the change is slow to take, you can force a reload by briefly moving the manifest out of the directory and back.

mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/
# Move it back a moment later
mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/

Cause B: cannot reach etcd #

apiserver only comes up if etcd is alive. If etcd is dead or apiserver’s --etcd-servers address is wrong, apiserver can’t attach to etcd and dies during startup. If you see connection refused or an etcd-related error in the logs, the root of the problem may be etcd, not apiserver. In that case, move on to the etcd diagnosis in Symptom 2 below.

Cause C: certificate problem #

apiserver uses certificates when it communicates with etcd and when it authenticates clients. If a certificate path is wrong or a certificate is expired, apiserver can’t come up. Check that the files pointed to by manifest flags like --client-ca-file, --etcd-cafile, and --tls-cert-file actually exist. Certificate expiry itself is covered separately in #25.

Symptom 2: etcd down #

When etcd dies, the cluster’s state store disappears. apiserver can’t attach to etcd and dies along with it, and as a result kubectl freezes too. In other words, a symptom that looks like apiserver down is often really caused by etcd. So when you see an etcd connection error in the apiserver logs, look at etcd first.

# etcd container state and logs
crictl ps -a | grep etcd
crictl logs <etcd-container-id>

There are two causes for etcd not coming up.

• Data directory problem. The directory pointed to by --data-dir (usually /var/lib/etcd) is corrupted, or the manifest’s hostPath points to the wrong path. There’s also the case where the disk is full and etcd can’t write.
• Certificate problem. etcd also communicates with its own certificates. If the certificate paths under /etc/kubernetes/pki/etcd/ are wrong or the files are missing, etcd can’t come up.

If etcd died from a manifest error, fix /etc/kubernetes/manifests/etcd.yaml to bring it back. If the data itself is corrupted and etcd can’t revive, this is the moment for the snapshot recovery you learned in #7.

Recovery when etcd data is corrupted #

If you have a snapshot, the recovery flow is identical to #7. Unpack the saved snapshot into a new data-dir, then change the etcd manifest’s hostPath to that directory so kubelet restarts etcd with the new data.

# 1) Restore the snapshot into a new directory (offline work, no certificates needed)
ETCDCTL_API=3 etcdctl snapshot restore /opt/snapshot.db \
  --data-dir=/var/lib/etcd-restore

# 2) Change the hostPath in /etc/kubernetes/manifests/etcd.yaml
#    from /var/lib/etcd to /var/lib/etcd-restore

# 3) Save and kubelet recreates etcd
crictl ps | grep etcd

The pitfalls of the recovery procedure (not passing certificates to restore, not forgetting to update the hostPath after restoring) are organized in #7 etcd Backup and Recovery, so I recommend getting it into your hands with that post before the exam.

Symptom 3: scheduler / controller-manager down #

When apiserver and etcd die, the whole cluster stops, so the symptom is dramatic. By contrast, when scheduler and controller-manager die, the cluster responds but only certain behaviors quietly stop. Because kubectl works normally, it’s actually harder to notice.

• kube-scheduler down. This is the component that decides which node to place a new Pod on. When it dies, newly created Pods stay stuck in Pending. Already-running Pods are fine. If Pending won’t clear in kubectl get pods and there are no scheduling-related messages in the events from describe, suspect the scheduler.
• kube-controller-manager down. This is the component that runs the reconciliation loop to converge toward the desired state. When it dies, self-healing stops. Increasing a Deployment’s replicas creates no new Pods, dead Pods aren’t recreated, and Pods aren’t rescheduled when a node drops out.

Diagnosis and recovery are the same as apiserver. Since both are static Pods, look at the relevant manifest and read the container logs to find the cause.

# scheduler / controller-manager container state
crictl ps -a | grep -E 'scheduler|controller'
crictl logs <container-id>

# manifests
cat /etc/kubernetes/manifests/kube-scheduler.yaml
cat /etc/kubernetes/manifests/kube-controller-manager.yaml

These two are relatively less urgent since the cluster doesn’t halt immediately when they die, but you have to remember that the root of symptoms like “a Pod won’t clear from Pending” or “scaling a Deployment gets no response” may be right here.

How to fix a static Pod: edit the manifest → kubelet restarts automatically #

All four components recover the same way. Fix the manifest and kubelet restarts on its own. To lay out the flow one more time:

1. Go into the control plane node and back up the relevant file in /etc/kubernetes/manifests/.
2. Read what the problem is with crictl logs and journalctl -u kubelet.
3. Fix the manifest’s typo, flag, path, or image and save.
4. kubelet detects the change and recreates the Pod. kubectl apply is not needed.
5. If the change is slow to take, force a reload by briefly moving the manifest out of the directory and back.

# When a forced reload is needed
mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/
# A moment later
mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/

There are two common mistakes here. One is reaching for kubectl apply or systemctl restart after fixing the manifest — a static Pod takes effect just by saving the file, so neither is needed. The other is skipping the backup before editing: if you introduce another error, you can’t get back to the original value. That one cp line before editing protects your time.

Symptom-to-cause diagnostic table #

This is a map for narrowing control plane troubleshooting from symptom to cause.

The order you read the table in is the order of diagnosis. If kubectl is frozen, look at apiserver; if the apiserver logs point to etcd, go down to etcd; if the etcd data is corrupted, move on to snapshot recovery. If kubectl is alive but only Pods are Pending or self-healing has stopped, suspect scheduler and controller-manager.

Exam points #

• All four control plane components are static Pods. Both diagnosis and recovery revolve around the manifests in /etc/kubernetes/manifests/.
• When apiserver dies, kubectl freezes. At that point, go into the node and look at the container and kubelet logs directly with crictl and journalctl.
• Common causes of apiserver down are manifest typos, can’t-reach-etcd, and certificate path errors. The logs tell you which one.
• Even when it looks like apiserver down, the real cause is often etcd. When you see an etcd connection error in the apiserver logs, look at etcd first.
• If the etcd data is corrupted, revive it with the snapshot recovery from #7. restore needs no certificates, and the key is not forgetting to update the hostPath after restoring.
• When scheduler dies, new Pods stay Pending; when controller-manager dies, self-healing stops. kubectl is normal, so it’s hard to notice.
• A static Pod restarts via kubelet just by saving the manifest. No kubectl apply or systemctl restart needed. Backing up before editing is mandatory.

Wrap-up #

What this post locked in:

• The single fact that the control plane runs as static Pods runs through all of diagnosis. Even if kubectl freezes, fixing the manifest on the node makes kubelet revive it.
• Diagnostic tools: when apiserver dies, instead of kubectl use crictl ps -a/crictl logs for containers, journalctl -u kubelet for kubelet, and the manifest for typos, flags, and paths.
• Cause by symptom: apiserver down (manifest/etcd/certificate), etcd down (data-dir/certificate), scheduler down (Pod Pending), controller-manager down (self-healing halted).
• Recovery: fix the manifest and save, and kubelet restarts. etcd data corruption is answered by the snapshot recovery in #7.

The instinct for reviving the control plane stands on top of the map “which component does what, and what stops when it dies” that we built up in #2 Cluster Architecture. Know the roles of the components exactly, and just by looking at a symptom you’ll immediately picture which manifest to go down to.

Next: Troubleshooting 4 #

Once you’ve revived the control plane, the cluster’s skeleton stands again. The last thing left is the connectivity and permissions layer. In #25 Troubleshooting 4: Networking, DNS, RBAC, certificate expiry, we’ll narrow down, symptom by symptom, problems where communication between a Service and a Pod is cut off, where CoreDNS failure breaks name resolution, where insufficient RBAC permissions get a request denied, and where certificate expiry makes apiserver and etcd no longer trust each other and cuts communication. It’s the last piece of the troubleshooting domain.