Sizing Servers for Self-Hosted LLMs — From VRAM Math to Concurrent Users

“How much server do we need to host an LLM in-house?” has no fixed answer, but it has a fixed order of calculation: compute VRAM from model size, add headroom from concurrency, then check bandwidth against the target speed. This post walks that order through to example configurations. It follows training versus inference and deals with the inference (serving) side. How LLMs work is covered in the explainer, and application development in the LLM app development series.

Step 1 — VRAM for the weights: parameters × bytes #

The starting point is simple multiplication: the memory the model weights occupy is parameter count × bytes per parameter.

Two practical instincts fall out. First, a BF16 70B doesn’t fit most single GPUs (not in an H100’s 80GB; it does fit a B200’s 192GB) — it needs tensor parallelism across cards, and at that moment the requirement jumps to NVLink-equipped hardware. Second, quantization rewrites the whole estimate. The same 70B at 4-bit drops to a 48GB-class card plus headroom. There is some quality loss, but for many internal uses it’s acceptable — so budget reviews are most useful as a two-line comparison: with and without quantization.

Step 2 — the KV cache: the memory concurrency eats #

Stop at the weights and you’ll hit an out-of-memory error on launch day. LLM serving keeps a KV cache per request — the attention keys and values of the conversation in progress — and it grows in proportion to concurrent requests × context length. Depending on model and settings, a few dozen concurrent long-context requests routinely consume tens of gigabytes on their own.

The working rules:

• Reserve 20–40% of VRAM beyond the weights for KV cache and runtime overhead. Putting 70GB of weights on an 80GB card breaks at a concurrency of 1–2.
• Set a context length cap as policy. Leaving “supports 128K” wide open lets a handful of long requests monopolize memory. A cap that fits the use case (say 16K–32K for internal document Q&A) is a precondition of any sizing.

Step 3 — speed and concurrency: bandwidth decides #

The last variable in GPU count is the speed target. Split it into two metrics: time to first token (TTFT) and time per output token (TPOT). The usual perceived-quality baseline: TTFT within a couple of seconds, and generation above 20–30 tokens per second (faster than reading speed).

Token generation, as the previous post covered, is governed by memory bandwidth. Simplified: a single request’s generation ceiling ≈ memory bandwidth ÷ weight size. With 3,000GB/s of bandwidth and 40GB of weights, the theoretical ceiling is in the seventy-tokens-per-second range, with real-world numbers below that. For concurrency, serving engines like vLLM use continuous batching to fill the GPU — total throughput rises, but per-request speed falls as concurrency climbs. When your user count doubles, the first move isn’t buying GPUs; it’s benchmarking for the concurrency ceiling at which per-request speed still holds the SLO.

Scenarios — three scales #

The math becomes intuition through scenarios. For picking specific GPUs, apply the VRAM and bandwidth criteria from the naming guide.

• Small internal tool (up to ~10 concurrent users, 8B-class model) — a 4-bit 8B is around 5GB, so one 24GB-class inference card (L4-class) covers weights and KV cache comfortably. One server; cloud costs start in the low hundreds of dollars a month. A flagship GPU at this scale is overkill.
• Mid-size service (tens of concurrent users, quantized 32B–70B) — a 4-bit 70B (~40GB plus cache) starts at two 48GB cards or one 80GB-class card. If tensor parallelism is involved, NVLink-equipped configurations pull ahead. With redundancy, two identical nodes behind a load balancer is the practical minimum.
• Large, high-quality (unquantized 70B+ at high concurrency) — tensor-parallel nodes of multiple 80GB-class cards, scaled horizontally. From here, the training-class infrastructure discussion — power, cooling, utilization — applies to serving too, and the break-even math below matters most before an investment of this size.

Before self-hosting — the break-even against APIs #

The final step of sizing is the sanity check: “is self-hosting actually cheaper?” Commercial APIs bill per token, favoring low usage; self-hosting is a fixed cost, favoring high and steady usage. Estimate monthly token volume, price it at API rates, and set that against GPU cost (cloud rental or purchase amortization, plus power and operations). In practice, the common justifications for self-hosting are data-egress restrictions, latency requirements, and custom models — not price. If cost is the only argument, redo the math with a conservative utilization assumption (how many hours a day does the GPU actually work?). An idle GPU is the most expensive form of waste.

The decision order #

1. Fix the model and quality bar — candidate model sizes and whether quantization is acceptable.
2. Compute VRAM — parameters × bytes, plus 20–40% for KV cache and headroom, giving card capacity and count.
3. Fix the speed requirements — TTFT and generation-rate SLOs, filtering candidate GPUs by bandwidth.
4. Benchmark — reproduce the real workload shape (context lengths, concurrency) on the candidate setup with vLLM or similar, and measure the concurrency ceiling. Spec math filters candidates; measurements make the decision.
5. Break-even check against APIs — compare at the estimated volume, and write down the non-cost requirements (data, latency, custom models) explicitly.

Summary #

• Sizing starts with multiplication: weight VRAM = parameter count × precision bytes, and quantization cuts that number to half or less.
• Beyond weights, reserve 20–40% of VRAM for the KV cache, and make the context length cap a policy decision.
• Generation speed is set by memory bandwidth; concurrency trades per-request speed for throughput via batching. Find the SLO-holding concurrency ceiling by measurement.
• Small scale runs on a single inference card; flagship GPUs only become necessary with unquantized large models and high concurrency.
• The last check is the API break-even. If cost is the only justification for self-hosting, distrust the utilization assumption.