Why Your Server Is Slow #4: Plenty of Bandwidth, Still Slow — Latency, RTT, Retransmissions

The link is 1Gbps, but API responses are slow. The bandwidth graph shows usage under 10%. Bandwidth to spare — so why is it slow? Hardware Basics #6 established the concept that bandwidth and latency are different axes; this post applies that distinction to a real diagnosis. Conclusion first: for traffic made of many small requests, perceived speed is set by round-trip count × round-trip time, not by bandwidth.

The formula for perceived speed — RTT × round trips #

Between Seoul and the US West Coast, RTT sits around 130ms — a floor set by the speed of light that no better circuit removes. On this path, one HTTPS request over a fresh connection stacks up round trips like this:

Almost 400ms gone, and barely a byte of payload moved. Raising bandwidth to 10Gbps shaves not one millisecond off that number. The only prescriptions are fewer round trips (connection reuse, TLS session resumption) or shorter round trips (moving regions, CDNs, edge).

The same structure repeats inside your own service. If handling one request makes 5 microservice calls and 10 database queries sequentially, then even a 1ms internal RTT gets multiplied by 15. Misplace a region so internal calls cross it, and the unit being multiplied jumps to tens of milliseconds. This is why batching and parallelizing sequential calls beats any bandwidth upgrade.

Connection reuse — the cheapest way to delete round trips #

The two round trips of TCP and TLS in that table disappear entirely when connections are reused, which is why reuse is always near the top of the checklist.

• Confirm the HTTP client actually uses keep-alive and a connection pool. Code that builds a new client object per request pays the handshakes every time.
• If a server or proxy idle timeout is too short, pooled connections keep dying and the reuse rate collapses.
• DNS lookups are round trips too. A TTL of 0 or a slow resolver adds tens of milliseconds per request.

Whether reuse is happening can be checked server-side from the ratio of new connections: if the TCP accept rate tracks the request rate closely, almost nothing is being reused.

When bulk transfers can’t fill the pipe — BDP and windows #

If the problem is instead large transfers — file copies, replication — then bandwidth is the right axis, but latency intrudes here too. TCP limits how much data can be in flight before an acknowledgment (the window), so one connection’s throughput ceiling is window size ÷ RTT. The window needed to fill a path is bandwidth × RTT — the BDP (bandwidth-delay product).

At 1Gbps × 130ms RTT, the BDP is about 16MB. If the window stalls at 4MB, one connection tops out around 250Mbps on this path — the classic shape of a mostly idle line with a slow transfer. For slow long-haul transfers, check that kernel TCP buffer ceilings (net.ipv4.tcp_rmem/tcp_wmem) aren’t smaller than the BDP; at the application level, splitting into parallel streams is the practical workaround.

Packet loss and retransmissions — a little goes a devastating way #

TCP reads loss as a congestion signal and cuts its sending rate immediately. So a loss rate that looks tiny — 0.1% — can slash throughput on long, fat paths by an order of magnitude or more, and each retransmission timeout puts a spike of hundreds of milliseconds on an individual request. It’s the usual culprit behind “fast most of the time, but it hiccups.”

Retransmissions are visible per socket:

$ ss -ti dst 10.0.3.7
ESTAB  0  0  10.0.1.21:44712  10.0.3.7:5432
     cubic rto:204 rtt:1.8/0.4 retrans:0/842 bytes_retrans:1218432

If the cumulative retrans count (842) and bytes_retrans keep growing, this path is losing packets. Narrow down where with per-hop loss in mtr. The cause is often equipment rather than line quality: queue drops on an overloaded switch, a bad duplex negotiation, or an MTU mismatch — path MTU discovery failing so that only large packets vanish. If small responses are fine but large ones stall, suspect MTU.

Measurement tools — measure each axis separately #

Diagnosis gets confusing because latency, bandwidth, and loss all reach you as one lump called “slow.” Split the axes and measure each.

• RTT and path: round-trip time with ping, per-hop latency and loss with mtr. Compare loaded and idle periods to catch queuing delay (bufferbloat) too.
• Bandwidth: measure the path’s actually achievable throughput with iperf3. The baseline is the measurement, not the circuit’s rated spec.
• Retransmissions: ss -ti per socket; server-wide, watch the TcpRetransSegs trend in nstat.
• Round-trip count: application tracing — how many internal calls one request fans into. This axis is invisible to system tools.

The diagnostic order #

When the symptom is “bandwidth to spare, but slow,” the order is:

1. Measure RTT first — ping and mtr. The physical distance of the path is the floor.
2. Count the round trips — new-connection ratio (wasted handshakes) and internal calls per request. Cutting the multiplication is the first prescription.
3. Check retransmissions — ss -ti and per-hop loss in mtr. If present, redirect toward equipment and path problems.
4. For bulk transfers, compute the BDP — check windows and buffers against it.
5. Whatever remains is the server — from this point it isn’t the network but the peer’s processing delay. Hand off to part 1’s diagnosis.

Summary #

• For small-request traffic, perceived speed is RTT × round-trip count. Bandwidth upgrades do nothing to that multiplication.
• Connection reuse (keep-alive, pools, TLS session resumption) deletes two round trips per request for free — the first prescription.
• One connection’s throughput ceiling is window ÷ RTT. For long-haul bulk transfers, check buffers against the BDP.
• Even 0.1% packet loss can cut throughput to a small fraction of the link’s capacity. Check with retrans in ss -ti and mtr.
• Latency, bandwidth, loss, and round-trip count are different axes. Measured separately, they yield a prescription.

The next part closes the series: the database. Why a database that ran fine keeps getting slower, traced through indexes, locks, and connection pools.