Where Kubernetes Gets Slow — API Server, etcd, Scheduler

Infrastructure Kubernetes Troubleshooting
4 min read

Service traffic is normal, but kubectl get takes seconds, rollouts crawl, and webhooks occasionally time out. This is the control plane getting slow — not the workloads (the data plane), but the part that steers the cluster. This post walks through the three places that bottleneck: the API server, etcd, and the scheduler. Pod-level troubleshooting was covered in the Pending post and the CKA troubleshooting series; this is the layer you meet after the cluster grows.

Separate the layers first. With a slow control plane, already-running Pods usually keep serving traffic just fine. Conversely, “the service is slow” belongs to the Why Your Server Is Slow series, not here. Take this post’s path when the symptoms cluster around kubectl, deployments, and controller reaction time.

1. The API server — who is hammering it? #

Every request passes through the API server, making it the first suspect. And what loads it down is usually not humans with kubectl but automated clients.

  • Repeated expensive LISTs — listing all Pods across namespaces without label filters is heavy on etcd and API server memory. Operators polling in loops, CI scripts, and monitoring agents are the usual offenders. A well-built controller does one LIST plus WATCH (informer cache); polling LISTs are what you fix.
  • Webhook chains — a slow admission webhook taxes every write request with its latency. If the webhook backend Pod sits on a struggling node or is restarting, cluster-wide writes slow down with it.
  • Check: apiserver_request_duration_seconds by verb and resource shows what’s slow; the apiserver_flowcontrol_* metrics (API Priority and Fairness) show who’s queued. With audit logging on, you can rank call volume by client.
  • Fix: convert polling clients to the informer pattern, add label selectors and pagination to LISTs, shrink slow webhooks’ timeouts and revisit their failurePolicy. APF’s built-in priorities mean one runaway client rarely takes the whole cluster down anymore — but that client, and everything in its priority band, still gets slow.

2. etcd — the control plane’s disk problem #

etcd is the cluster’s only store, and it fsyncs every write. So etcd performance problems are mostly disk problems — the fsync story from the SSD post makes its return here.

  • fsync latency — etcd’s guidance is a p99 fsync in single-digit milliseconds. If etcd_disk_wal_fsync_duration_seconds runs above that, look at storage-side causes: network volume IOPS ceilings, noisy neighbors on shared disks, devices without PLP. The “apply entries took too long” warnings in etcd logs are the same signal.
  • Database bloat — many objects and frequent changes (Events especially, plus operators that update CR status constantly) pile up revisions; the DB grows, slows as it grows, and at the quota (2GB by default) the cluster drops to read-only — a real incident pattern. Track etcd_mvcc_db_total_size_in_bytes over time, confirm compaction runs, and defragment when needed.
  • Large objects — giant ConfigMaps (whole file bundles) and oversized CRD objects weigh down both writes and watches. Data files belong in object storage.
  • Fix: dedicated low-latency disks for etcd (local SSD preferred), event TTLs and slimming down objects, periodic compaction and defrag. On managed clusters (EKS and friends) etcd is the provider’s job — but keeping object counts and change rates low remains yours.

3. The scheduler — a long queue or heavy math #

Scheduler delay shows up as time between Pod creation and node assignment. Two shapes:

  • A long queue — a batch job creating thousands of Pods at once backs up the scheduling queue. That’s natural queueing more than a bottleneck; pacing batch creation is the fix.
  • Expensive per-Pod computation — on clusters with thousands of nodes, complex pod affinity/anti-affinity (especially topology-based) costs real evaluation time. If scheduler_scheduling_attempt_duration_seconds stretches, the standard fix is replacing non-essential anti-affinity with topology spread constraints.

A slow scheduler and nowhere to schedule (Pending) are different problems: the former eventually assigns, the latter leaves FailedScheduling events.

The diagnostic order #

  1. Separate from data-plane problems — do the symptoms cluster around kubectl, deploys, controllers? If service responses are slow, that’s another post’s territory.
  2. API server metrics — which verb/resource is slow (apiserver_request_duration_seconds); find runaway clients via APF metrics and audit logs.
  3. etcd metrics — fsync p99 and DB size trend. Disk is most of it.
  4. Webhooks — writes-only slowness points to the admission chain.
  5. Scheduler — for assignment lag, check queue length and attempt duration, and clean up heavy affinity.

Summary #

  • Control-plane slowness is a separate layer from workload traffic. First check whether symptoms cluster on kubectl, deploys, and controllers.
  • What weighs the API server down is mostly polling LISTs from automation and slow admission webhooks. Informers, selectors, and pagination are the fixes.
  • etcd problems are mostly disk (fsync) problems; the rest is object bloat. Single-digit-ms fsync p99 and DB size trend are the metrics to hold.
  • The scheduler suffers from mass-creation queues and heavy affinity math; topology spread constraints are the standard replacement.
  • Even on managed clusters, object counts, change rates, and webhook quality stay your responsibility.
X