AI Training Servers vs. Inference Servers — Where the Requirements Diverge

Infrastructure Hardware AI GPU
5 min read

Shop for GPU servers and you’ll find one source describing an eight-GPU NVLink node costing as much as a house, and another saying a single card for a few thousand dollars will do. Both are right — the first is about training, the second about inference. This post covers why the two demand different machines, and exactly where the requirements diverge. It follows the GPU server basics and how to read GPU names.

The workload shapes differ #

Training is a batch job. It streams massive data through the model repeatedly, updating weights, holding GPUs at 100% for days to weeks. The goal is total throughput — “does it finish in days or weeks?” The concept of per-response latency doesn’t exist.

Inference is request serving. A user request arrives and an answer goes back; traffic swells and ebbs. The goal is serving within a latency SLO — “does the first token appear within seconds, how many requests per second?” This is the side where the instincts from the Why Your Server Is Slow series carry over directly.

The batch-versus-serving difference produces every requirement difference below.

The bottleneck differs — interconnect versus memory #

Training bottlenecks on GPU-to-GPU communication. Large models don’t fit on one GPU, so dozens to thousands of them shard the model and data — and exchange gradients every step. When that exchange is slow, GPUs sit idle waiting on the network. That’s why training nodes make NVLink within the node and InfiniBand/RoCE between nodes the heart of the spec, and why flagship (H100/B200-class) nodes cost what they do. Checkpoint read/write storage bandwidth also matters mostly on this side.

Inference usually bottlenecks on memory. Generating each token requires reading the entire model’s weights from VRAM, so token throughput is governed more by memory bandwidth than by compute. The requirement collapses to “does the model fit in VRAM, and does the bandwidth deliver the target speed” — and GPU-to-GPU interconnect isn’t needed at all as long as the model fits on one card. This is why NVLink-less inference cards (L4, L40S) exist as a category.

Precision differs — quantization, inference’s weapon #

Training needs relatively high precision (BF16 as the default; recent generations also train in FP8) for numerically stable gradients. Inference only reads frozen weights, so quantization — dropping to INT8, FP8, or 4-bit — works with acceptable quality loss.

The implication for hardware selection is significant. Quantization halves or quarters the VRAM and bandwidth requirements, so the same model serves on far smaller GPUs than it trained on. Training a 70B model takes a flagship node; serving a 4-bit-quantized 70B works on a couple of mid-range GPUs. The concrete math is in the next post.

The scaling direction differs — up versus out #

  • Training scales up, then out in tight coupling. The communication bottleneck demands GPUs stay close — one NVLink domain, one InfiniBand fabric — and adding nodes makes the network design harder. That’s why rack-scale products like the GB200 NVL72, sold as one NVLink domain, exist.
  • Inference scales out loosely. Work divides at the request level, so multiple model-loaded servers behind a load balancer scale horizontally. Autoscaling with traffic only exists on the inference side.

Availability splits the same way: a dead training node means restarting from a checkpoint — it’s a batch job. Inference is user-facing and needs the same redundancy, health checks, and deployment strategies as any web service.

The cost structures differ #

  • Training cost is close to a lump sum. Heavy use during a project, then done — a good fit for reserved or spot capacity and GPU cloud rentals, with spot interruptions absorbed by checkpoint restarts. Owning training hardware is justified mainly where pipelines run year-round.
  • Inference cost is a fixed cost. It runs as long as the service lives, and idle capacity at low-traffic hours is pure waste. So inference practice centers on utilization management: fill the GPU with batching, trim idle with autoscaling, and if traffic is small, check whether per-token API pricing is simply cheaper.

The trap of a shared fleet #

“Let’s also serve on the training servers” is a natural thought with two traps. First, waste: you paid for NVLink and InfiniBand that inference doesn’t use, while inference performance is still capped by memory bandwidth. Second, interference: while training occupies the GPUs, inference latency blows through its SLO. If you must share, partition GPUs with MIG (Hardware Intermediate #8), split by time window, or — simplest to operate — separate the pools from the start.

Summary #

  • Training is a batch job (total throughput); inference is request serving (latency SLO). That shape difference is the root of every spec difference.
  • Training bottlenecks on GPU interconnect, making NVLink/InfiniBand the core of the spec; inference bottlenecks on VRAM capacity and memory bandwidth.
  • Quantization slashes inference requirements, so the same model serves on much smaller GPUs than it trained on.
  • Training scales in tight coupling (rack-scale); inference scales horizontally behind a load balancer with autoscaling.
  • Training cost is episodic (spot and rentals win); inference cost is fixed (utilization management is the game). Shared fleets invite waste and interference — separate pools by default.

The next post turns this difference into numbers: estimating the server scale needed to self-host an LLM, starting from the VRAM math.

X