LLM API vs Self-Hosting: Doing the Break-Even Math

Infrastructure AI Cost
5 min read

As open-weight models became runnable in-house, a question appeared: is it better to keep using an API, or to grab a GPU and self-host? The answer comes from arithmetic, not taste. The conclusion up front: an API is right for most teams, and self-hosting is the choice when steady throughput keeps a GPU full or when privacy and regulatory requirements are explicit. Server sizing itself is covered in LLM Self-Hosting Server Sizing; this post is the “should we do it at all” calculation that comes before it. Prices are as of mid-2026.

Billing structure: per token vs per GPU-hour #

An API bills per token. It prices input and output tokens in units of a million, and costs nothing when idle. A mid-tier model (say, Llama 3.3 70B serverless) runs around $0.88 per million tokens each way; frontier commercial models (Claude Sonnet, GPT, Gemini Pro class) run $2〜3 input and $12〜15 output. Infrastructure, scaling, and upgrades are all baked into the price.

Self-hosting bills per GPU-hour. However many tokens you produce, you pay for the time you hold the GPU. One H100 runs $4〜8 an hour on hyperscaler on-demand, $2〜3.3 on specialist GPU clouds (Lambda, RunPod, and the like), and down to $1〜2 on spot. Idle time still costs money as long as the GPU is held.

So the comparison narrows to one axis: the fraction of time the GPU actually produces tokens — utilization.

Break-even: the tokens one GPU produces in a day #

Run the numbers. Put a 70B-class model on a single H100 with vLLM at 4-bit quantization (see the quantization comparison), batch concurrent requests, and you get throughput in the hundreds to a thousand tokens per second. Take a comfortable 500 tokens/second: that is 1.8M tokens per hour, and about 43M tokens over a full 24 hours.

  • Self-hosting cost: an H100 at a specialist cloud’s on-demand $3/hour is $72 a day. Producing 43M tokens works out to about $1.7 per million tokens. At spot $1.5/hour it drops to about $0.8 per million.
  • API rate: an open model of the same class runs around $0.88 per million tokens serverless.

The key emerges here. Only by keeping the GPU full 24 hours does self-hosting cost approach or slightly undercut the API. At half utilization the per-token cost doubles and the API clearly wins. A service that gets traffic only a few hours a day is buying an idle GPU by self-hosting. Conversely, a workload that saturates the GPU continuously with batch and bulk processing (mass document classification, embedding generation, offline summarization) crosses over into economies of scale.

The variables that move the crossover #

  • Utilization: this is the whole calculation above. Full 24 hours favors it, intermittent works against it. The always-on GPU with no scale-to-zero is self-hosting’s intrinsic weakness.
  • Batch processing: work that needs no real-time response can fill the GPU to 100% via batching. APIs offer batch discounts (about 50%) too, but self-hosting lowers per-token cost directly by driving utilization up.
  • Quantization: dropping to 4-bit fits a bigger model on the same GPU or raises throughput, lowering per-token cost — at the price of some accuracy, so be careful with reasoning and code tasks (see the quantization comparison).
  • Model class: replacing a frontier commercial model (output $15/million) is a different calculation from replacing a mid-tier open model. If you need frontier-class quality and open models cannot deliver it, self-hosting is not an alternative to begin with.

Hidden costs: the GPU rate is not the whole bill #

Self-hosting cost carries items beyond the GPU-hour rate. The MLOps people who deploy and update models, redundancy for availability (a second GPU), handling traffic spikes, and re-validation every time a new model drops all become standing work. An API absorbs all of it into the vendor. The gap between demo and production is covered in LLM App Operations #1; self-hosting stacks a layer of infrastructure operations on top of that.

APIs have hidden items too. Skip prompt caching (about 90% off repeated prompts) and the bill inflates; there are rate limits, vendor lock-in, and the risk of a model version changing without notice.

What decides it outside of price #

When break-even is ambiguous, the decision comes from requirements, not price.

  • Data privacy and regulation: in medical, financial, or classified environments where data cannot leave for an external API, it is self-hosting (or a dedicated in-VPC deployment) regardless of cost.
  • Latency: an on-prem GPU that is physically close eliminates the network round trip. Though a regional API endpoint narrows much of that gap.
  • Customization: serving fine-tuned weights directly or applying special decoding is freer when self-hosted.
  • Team capacity: without people to operate GPU serving, self-hosting is effectively more expensive even when the raw cost is a little lower.

The selection order #

  1. Filter on requirements first: a regulation against sending data outside means self-hosting, full stop; otherwise move to the math.
  2. Estimate utilization: steady or batch throughput that saturates a GPU 24 hours makes self-hosting a candidate; intermittent traffic means an API.
  3. Match the model class: first confirm an open model meets your quality bar. If frontier class is required, self-hosting is not an alternative.
  4. Compare on total cost: add redundancy, operations staff, and re-validation to the GPU-hour rate, and compare against the API’s post-caching rate.
  5. Consider hybrid: starting on an API and peeling off only batch bulk work to self-hosting is a common optimum.

Summary #

  • APIs bill per token, self-hosting per GPU-hour. The comparison narrows to a single axis: GPU utilization.
  • Only a fully utilized 24-hour H100 brings self-hosting cost (about $1.7 per million tokens, $0.8 on spot) into contention with an open-model API (around $0.88). At low utilization the API clearly wins.
  • Batching and quantization raise utilization and throughput, moving the crossover toward self-hosting.
  • Operations staff, redundancy, and re-validation beyond the GPU rate are self-hosting’s real cost. The API’s hidden items are un-applied caching and vendor lock-in.
  • When break-even is close, privacy, regulation, latency, and team capacity decide. Default API plus self-hosting only batch work is a common optimum.
X