Why Your Server Is Slow #3: Slow Despite an SSD — Write Amplification, fsync, Queue Depth

Infrastructure Hardware Performance
6 min read

There’s a common instinct that swapping an HDD for an SSD fixes everything. Yet on a server with an NVMe SSD, following part 1’s diagnosis can land you at iostat showing await in the tens of milliseconds. The spec sheet says hundreds of thousands of IOPS — why doesn’t this SSD behave like its spec?

The skeleton of the answer is one sentence: spec-sheet numbers are maximums under specific conditions, and real workloads rarely satisfy those conditions. The things that break the conditions are this post’s topics: fsync, write amplification, and queue depth. Reading device specs is covered in Hardware Basics #4 and measurement methodology in Hardware Intermediate #5; this post focuses on where the gap between spec and reality comes from.

fsync — the moment you demand durability, it’s a different game #

An ordinary write is finished once it lands in the page cache, so it returns at memory speed. But writes that must survive a power cut — a database commit, a message queue journal — go through fsync, which pushes the data down to the device and waits for confirmation. That path has to bypass or flush the SSD’s internal cache too, so the same device produces latencies in a different order of magnitude.

Consumer SSDs in particular often have an extremely slow fsync path. A datacenter SSD with power-loss-protection capacitors (PLP) can safely report completion the moment data hits its internal cache, so fsync finishes in tens of microseconds. Without PLP, the drive must actually commit to flash, and a single fsync can take milliseconds. This is most of the reason a consumer SSD that tops sequential-read benchmarks turns disappointing the moment you put a database on it. If the workload needs thousands of commits per second, fsync latency is commit latency.

You can measure fsync performance directly with fio.

fio — measuring fsync latency 
$ fio --name=fsynctest --rw=write --bs=4k --size=1g --fdatasync=1
  ...
  fsync/fdatasync/sync_file_range:
    sync (usec): min=280, max=18400, avg=1240.52

An average of 1.2ms puts this device’s synchronous commit ceiling around 800 per second — a number that has nothing to do with the spec sheet’s “500K write IOPS.”

Write amplification — built to slow down with age #

SSD flash cannot be overwritten in place. Erases happen per block; writes happen per page, which is smaller. So the controller writes new data to empty pages, marks the old pages invalid, and later runs garbage collection: moving the still-valid pages to another block so the old block can be erased. When you write 1, the device internally writes several times that, relocation overhead included — this is write amplification.

Amplification grows when empty blocks run short. Fill the disk close to capacity, or lose TRIM (so the filesystem’s deleted space still looks like valid data to the SSD), and GC has more valid pages to relocate — every write drags internal copying costs behind it. One more layer: many TLC/QLC SSDs absorb writes into a fast SLC-mode cache first, and sustained writes exhaust it, dropping speed in a visible step. “The copy starts fast and falls off a cliff after tens of seconds” is the signature symptom.

There are three operational prescriptions: keep usage below 80–90% so the controller has spare blocks, confirm periodic TRIM is actually running (fstrim.timer), and if sustained write performance matters, choose devices by their sustained-write figures and PLP — not the burst maximums.

Queue depth — spec IOPS requires a long line #

The spec sheet’s “1M random read IOPS” carries fine print: QD32, meaning queue depth 32, across multiple workers. An SSD is a parallel device with dozens of internal channels, and only a deep queue keeps all the channels busy.

But much of what real applications do is close to QD1. Walking down a B-tree index, following a pointer chain — any access where you need the previous result to issue the next request cannot form a line. At QD1 the deciding factor is not parallelism but the round-trip latency of a single request, and even top NVMe devices rarely get far below tens of microseconds. Converted to QD1 4K random reads, every SSD converges to the same tens-of-thousands-of-IOPS territory.

So “spec says 1M IOPS, we measured 20K” may not be a fault — the workload’s queue may simply be shallow. If aqu-sz (average queue length) in iostat -x hovers around 1, the device is idling below its capability. Upgrading to a faster device buys little here; the prescriptions are application-side parallelism, better access patterns, or absorbing those lookups into a cache in the first place. Measuring across queue depths is covered in Hardware Intermediate #5.

Cloud volumes — the ceiling is the contract, not the device #

In the cloud there is one more layer. Network block storage like EBS has IOPS and throughput ceilings set per volume and per instance — by contract. A gp3 volume defaults to 3,000 IOPS, so if you run it on physical-NVMe instincts, await spikes long before the numbers you’d expect. Volume types with burst credits (gp2 and friends) drop performance in a step the moment credits run out, producing “it was fine for days, then suddenly slowed down.” Check first whether the volume’s IOPS consumption is pinned at its ceiling; if so, the prescription is a volume spec change, not device tuning.

The diagnostic order #

When the symptom is “it’s an SSD, but storage is the bottleneck,” the order is:

  1. iostat -x — shape the symptom with await, aqu-sz, and the read/write mix.
  2. In the cloud, ceilings first — is IOPS or throughput pinned at the contract limit, or are burst credits gone? The most common cause and the fastest to check.
  3. Is it an fsync workload? — if database or journal writes dominate, measure synchronous write latency with fio --fdatasync=1. On a consumer device, the fix is a device with PLP.
  4. Does it slow down under sustained writes? — fast at first means SLC cache exhaustion and write amplification. Check free capacity and TRIM.
  5. Is aqu-sz around 1? — a shallow queue means access patterns and caching, not a device upgrade.

Summary #

  • Spec-sheet IOPS are maximums under deep queues and parallel workers. A measurement that differs from spec reflects different conditions, not a fault.
  • fsync writes are a separate game. Consumer SSDs without PLP get slower by orders of magnitude on commit workloads.
  • Write amplification grows out of empty-block scarcity. Watch capacity headroom, TRIM, and sustained-write specs.
  • Dependent accesses run at QD1, where latency is everything. With a shallow queue, device upgrades pay little.
  • Cloud volume ceilings arrive before device limits. Check the contract before tuning.

Next is the network: why things are slow when bandwidth is plentiful, traced through latency, RTT, and retransmissions.

X