When a Pod Is Stuck in Pending — 7 Causes and the Order to Check Them

Infrastructure Kubernetes Troubleshooting
5 min read

You deploy, and the Pod won’t move past Pending. Pending isn’t an error — it means “no node has been chosen yet” — so there are no logs (the container hasn’t started, kubectl logs comes back empty) and the cause isn’t immediately visible. This post lists the causes of Pending in order of real-world frequency. Exam-style troubleshooting in general is covered in CKA #22; here we focus on the scenarios you run into in operations.

Checking starts in one place — Events #

The scheduler almost always writes the reason for Pending into Events.

kubectl describe pod 
$ kubectl describe pod api-server-7d4b9c-x2k8p
...
Events:
  Type     Reason            Message
  ----     ------            -------
  Warning  FailedScheduling  0/6 nodes are available: 3 Insufficient cpu,
                             2 node(s) had untolerated taint {gpu: "true"},
                             1 node(s) didn't match Pod's node affinity/selector.

This message is close to the whole diagnosis: “0 of 6 nodes available” plus the breakdown (3 short on CPU, 2 tainted, 1 affinity mismatch). Treat the list below as the dictionary for interpreting it.

1. Insufficient resources — Insufficient cpu/memory #

The most common cause. The scheduler computes space from the sum of requests, not actual usage. Insufficient cpu means “on every node, the remaining allocatable is smaller than this Pod’s requests.”

  • Check: kubectl describe node shows Allocated resources per node — how full the requests total is.
  • Fix: more nodes or evicting other workloads is the direct fix. But it’s worth suspecting the next item first.

2. Oversized requests — usage is low but the reservations are full #

If kubectl top node shows 30% real usage but you still get Insufficient cpu, someone’s requests are set far above what they use. Requests are reservations — they hold the space whether it’s used or not. Enough “just to be safe” requests lock half the cluster in empty reservations.

  • Check: put per-workload requests and actual usage side by side (metrics-server, or VPA in recommendation mode).
  • Fix: re-baseline requests from measurements. How to set them, building on the concepts in K8s Intermediate #4, gets its own post.

3. nodeSelector and affinity mismatches #

didn't match Pod's node affinity/selector means no (or not enough) nodes carry the labels the Pod demands. Node labels changed or were removed, or the manifest has a label typo (disktype: ssd vs diskType: ssd) — those are the typical causes.

  • Check: compare kubectl get nodes --show-labels against the manifest’s requirements.
  • Fix: align the labels, or relax required affinity to preferred where the constraint isn’t truly mandatory.

4. No toleration for a taint #

untolerated taint is the node side declaring “not just anyone.” GPU nodes, dedicated node pools, and the default taint on control-plane nodes are the usual cases. If only GPU nodes have room left, every ordinary Pod in the cluster goes Pending — a classic incident.

  • Check: kubectl describe node <node> | grep Taints shows them.
  • Fix: if the dedicated nodes are intentional, secure capacity for general workloads; if the Pod belongs on those nodes, add the toleration.

5. The PVC won’t bind #

A Pod that mounts volumes schedules only once its PVCs are ready. waiting for a volume to be created, or a PVC sitting in Pending, points here. StorageClass name typos, a missing provisioner, and WaitForFirstConsumer mode getting entangled with other constraints are common. In multi-AZ clouds, the zonal mismatch — “the volume lives in zone a, but the only schedulable nodes are in zone b” — is a regular occurrence.

  • Check: kubectl get pvc and the Events in kubectl describe pvc.
  • Fix: verify the StorageClass, and rethink zone spreading — design affinity for volume-bearing workloads together with zones.

6. ResourceQuota and LimitRange violations #

With a quota on the namespace, Pods are refused even when nodes have room. The Pod may not even be created (the record lands in ReplicaSet Events), or you’ll see an exceeded quota message. Copy-pasted default quotas during new-team onboarding cause many of these.

  • Check: kubectl describe quota -n <ns> and the Events on the ReplicaSet or Deployment.
  • Fix: adjust the quota, or shrink the workload’s requests.

7. Cluster Autoscaler won’t scale up #

The case where the “autoscaler will add nodes” assumption betrays you. The node group hit its maximum, the cloud ran out of that instance type or the account hit a limit, or — subtle one — the Pod’s own constraints (affinity, taints) mean a new node wouldn’t be schedulable anyway, so the autoscaler declines. It records its reasons in events and logs.

  • Check: the NotTriggerScaleUp event in kubectl describe pod, and the autoscaler logs.
  • Fix: raise the node group ceiling, diversify instance types, or relax the Pod’s constraints.

The diagnostic order #

  1. kubectl describe pod — read Events first. The FailedScheduling message gives you the distribution of causes.
  2. If Insufficient, check reservation saturation in describe node’s Allocated resources, and compare with actual usage to find oversized requests.
  3. If selector/taint, compare node labels and taints against the manifest.
  4. For volume workloads, check PVC state and zone placement.
  5. On quota messages, check the namespace quota; with an autoscaler, check the NotTriggerScaleUp events.

Summary #

  • Pending isn’t an error but “nowhere to go,” and the reason is almost always written in describe’s FailedScheduling events.
  • The scheduler computes with request sums, not actual usage. Low usage with no room means suspect oversized requests first.
  • Label and taint mismatches fall out of a direct comparison between manifests and node state.
  • Volume Pods need PVC state and zone placement checked; autoscaler setups need the “why didn’t it scale” events read too.
  • If the same message keeps recurring, the root fix is request sizing standards and node pool design, not another one-off patch.
X