Why Your Server Is Slow #1: Low CPU Usage but Still Slow — Run Queue, I/O Wait, Lock Contention

Infrastructure Hardware Performance
7 min read

The CPU gauge on your monitoring dashboard reads 20%, yet response times keep stretching. The CPU has plenty of headroom — so why is the server slow? Every operator runs into this eventually, and it is where this series starts.

The series has five parts. Each one takes the symptom “the specs are fine, but it’s slow” and chases it down for a different resource: low CPU usage but slow (part 1), adding memory doesn’t help (part 2), slow despite an SSD (part 3), plenty of bandwidth but the network is slow (part 4), and a database that keeps getting slower (part 5). Where the Hardware Basics and Intermediate series build the concepts and metrics, this series works case-first: start from the symptom and descend to the cause.

What utilization doesn’t tell you #

The first thing to check is the question itself. “CPU usage is low” does not guarantee “the CPU is not the bottleneck.” Utilization is an average over an interval, and it only measures the share of time a task occupied a core. Two things slip through.

  • Bursts buried in the average — a 1-minute average of 20% is indistinguishable from 5% for 59 seconds and 100% for one. The queuing that happens in that one second never shows up on the graph.
  • Waiting that happens off the CPU — time a process spends waiting for disk I/O, a lock, or another service’s response uses no CPU, so utilization never sees it. Very often, most of the response time users feel lives in exactly this waiting.

So the diagnostic question changes from “how busy is the CPU” to “where are requests waiting.” There are three main places: the line in front of the CPU (the run queue), disk responses (I/O wait), and locks.

The run queue — the line in front of the CPU #

Threads that are ready to run but waiting for a free core form the run queue. That line can be long even when utilization is low: if short tasks arrive in bursts, each task queues up and gets delayed, while the average utilization barely moves.

The first column of vmstat is exactly this line.

vmstat 1 
$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
12  0      0 811240 219880 5480032    0    0     0    24 9821 21453 18  4 78  0  0
14  0      0 810988 219880 5480040    0    0     0     0 10233 22871 19  5 76  0  0

r is the number of threads waiting to run. This server sits around 23% utilization (us+sy), yet r is well above the core count (say, 8). When the line stays longer than the number of cores, scheduling delay lands directly on response time. The load average from Hardware Intermediate #1 gives a similar signal, but it includes tasks waiting on I/O, so for separating causes it’s better to read vmstat’s r and b (I/O wait) columns separately.

To measure the queuing time itself, the eBPF tool runqlat prints the distribution of scheduling delay as a histogram — the tool is covered in Hardware Advanced #2. If utilization is low but runqlat’s tail reaches into milliseconds, suspect bursts or far more runnable threads than cores. An application with a thread pool sized way beyond the core count is the classic case.

I/O wait — idle in disguise #

top’s %wa (iowait) is time when there is nothing runnable and at least one disk I/O is outstanding. Two traps live here.

  • iowait is a form of idle. It is not part of CPU utilization, so a server at “20% usage + 40% iowait” looks relaxed if you only watch the usage graph.
  • Conversely, low iowait does not mean no I/O bottleneck. If other tasks keep the CPU busy, the same I/O waiting is no longer counted as iowait.

So treat iowait only as a starting signal — “high, go look at the disks” — and make the verdict with disk-side metrics: await in iostat -x (average time one I/O took, queueing included) and %util, then narrow down the process with pidstat -d.

iostat -x 1 
$ iostat -x 1
Device            r/s     w/s   rkB/s   wkB/s  await  %util
nvme0n1         210.0  1830.0   840.0 29280.0   8.42   96.4

If a low-utilization server shows await at several times its normal level with %util above 90%, the bottleneck is storage, not CPU. Why those numbers happen even on an SSD is part 3’s topic.

Lock contention — waiting that is neither CPU nor disk #

If the run queue is short and the disks are quiet but it’s still slow, the remaining suspect is processes waiting on each other: mutexes, database row locks, a free connection in the pool, an upstream API’s response. These waits use no CPU at all, so they leave almost no trace in system metrics. The threads are asleep, utilization is low, and only the responses are slow.

The one system-side clue is context switches. A thread that keeps failing to get a lock sleeps and wakes over and over, so vmstat’s cs (context switches) runs abnormally high for the load. pidstat -w shows voluntary context switches (cswch/s) per process, which narrows down who keeps going to sleep waiting for something.

pidstat -w 1 
$ pidstat -w 1 -p 4321
UID       PID   cswch/s nvcswch/s  Command
 1001     4321   8412.0      12.0  api-server

Eight thousand voluntary switches per second means this process is not working and pausing — it is waiting and waking, repeatedly. From here the problem is inside the application, so you descend to off-CPU analysis with a language-level profiler to see which lock the threads are parked on. The two most frequent culprits in practice are connection pool exhaustion and long transactions holding locks behind a slow query — both of which part 5 revisits from the database side.

In the cloud — check steal time #

On a virtual machine there is one more suspect. top’s %st (steal) is time when the guest was ready to run but the hypervisor didn’t give it a physical core. Your server’s utilization is low, but another tenant on the same physical host is taking the cores — and your process slows down for no visible reason. If %st stays above a few percent, consider migrating the instance or changing the instance type. Burstable instances (the t families) produce the same symptom when CPU credits run out. Details are in Hardware Intermediate #2.

The diagnostic order #

When the symptom is “CPU usage is low but it’s slow,” the order is:

  1. Remove the averaging trap first — narrow the monitoring interval (down to 1 second) and check for bursts.
  2. vmstat 1 — if r exceeds the core count, suspect scheduling delay (too many threads, bursts); if b builds up, look at I/O.
  3. iostat -x — judge storage bottlenecks with await and %util. If it’s guilty, move to part 3’s territory.
  4. pidstat -w — high voluntary context switches point to locks, connection pools, or upstream waits; descend into application profiling.
  5. On a VM, check %st — steal time and CPU credits.

Summary #

  • CPU utilization is just an average of core occupancy. It doesn’t tell you where requests wait. The diagnostic question is not “is the CPU busy” but “where is the waiting.”
  • If the run queue (r in vmstat) stays above the core count, scheduling delay lands on response time even at low utilization.
  • iowait is a form of idle — it’s not in the usage number, and a low value doesn’t clear the disks. The verdict comes from iostat’s await and %util.
  • Locks, connection pools, and upstream waits barely register in system metrics. The clue is high voluntary context switches; the confirmation is off-CPU analysis.
  • On virtual machines, steal time and CPU credits produce the same symptom.

Next up is memory: why adding more of it didn’t make anything faster, traced through the page cache, swap, and the working set.

X