vLLM vs Ollama vs SGLang: Choosing an LLM Serving Framework

Infrastructure AI
4 min read

Once you have decided to self-host (see the break-even math), the next question is what to serve with. Put the same model on the same GPU and throughput differs several-fold by framework. The conclusion up front: laptops and local experiments go to Ollama, multi-user production defaults to vLLM, and complex prompt structures or agent workloads warrant a look at SGLang. Server sizing is covered in LLM Server Sizing; this post is the choice of software that runs on top of it.

Two branches: local execution vs production throughput #

Frameworks split into two branches with different purposes: “run it easily on one machine” and “serve many users at high throughput.” Mixing the two is where the choice goes wrong.

  • Ollama is a local execution tool built on llama.cpp. ollama pull fetches a model and an OpenAI-compatible API comes up locally right away. It is the simplest to install and use, and runs in VRAM-constrained environments via CPU+GPU hybrid execution. But it has no continuous batching, no PagedAttention, and no multi-node support. It is not built to serve concurrent users at high throughput.
  • vLLM is an engine built for production throughput. PagedAttention and continuous batching deliver 3〜4x the throughput of Ollama on the same GPU. It is effectively the standard for multi-user serving.
  • SGLang is in the same high-throughput family as vLLM, with strengths in complex prompt structures (branching, multi-step, structured output) and prefix caching. There are regions where it wins on agent and composite-pipeline workloads.

Why throughput diverges: PagedAttention and continuous batching #

Two techniques put vLLM and SGLang well ahead of Ollama.

  • Continuous batching: instead of waiting for a batch to finish, new requests are inserted into the running batch at every generation step. Because LLM requests vary in length, when a short one finishes and frees a slot it is filled immediately with a new request. GPU idle time shrinks.
  • PagedAttention: the KV cache is managed in pages like an operating system’s memory. Fragmentation drops, so more concurrent requests fit in the same VRAM.

In numbers, benchmarks commonly show vLLM in the hundreds of tokens per second and Ollama in the tens. The gap widens as load rises and narrows at light loads around 5〜10 concurrent users. That is, any of them works for a handful of users, but vLLM-class engines become mandatory as users grow. This throughput is what sets the per-token cost in the break-even math.

The 2026 shift: TGI enters maintenance mode #

Hugging Face’s TGI (Text Generation Inference), long cited as a vLLM alternative, entered maintenance mode in December 2025. It now accepts only bug fixes and documentation PRs, and Hugging Face recommends vLLM or SGLang for new Inference Endpoints. If an older comparison places TGI as a primary option, that part is stale. Today’s production trio is effectively vLLM, SGLang, and the special-purpose TensorRT-LLM (optimized for the NVIDIA stack).

The feature axis: quantization, structured output, compatibility #

Beyond throughput, these axes decide in practice.

  • Quantization support: vLLM broadly accepts the major methods — AWQ, GPTQ, FP8 (see the quantization comparison). Ollama is GGUF-centric. Which quantized file you plan to serve can itself narrow the framework choice.
  • Structured output: for JSON-schema enforcement or grammar-constrained decoding, both SGLang and vLLM support it but at different maturity. For workloads with strict output formats, like agent tool calls, check this first.
  • API compatibility: all three offer an OpenAI-compatible API, so you can switch from an API to self-hosting with no client code changes. This is why the hybrid transition from the break-even math is easy.
  • Operational maturity: vLLM has a large community and fast new-model support. If you need to serve a model right after release, support speed becomes a practical criterion.

The selection order #

  1. Filter on purpose first: local experiments, prototypes, or single-user use are fine on Ollama. Multi-user production means a vLLM-class engine.
  2. Count the throughput requirement: any option works for a few concurrent users, but if you plan to grow, start on vLLM to avoid a rewrite.
  3. Look at workload shape: simple chat and completion favor vLLM; agent pipelines heavy on branching, multi-step, and structured output warrant comparing SGLang.
  4. Match the quantized file: confirm the model’s quantization format (GGUF vs AWQ/GPTQ) matches framework support.
  5. Filter out stale options: avoid adopting TGI new (maintenance mode), and consider TensorRT-LLM only when you need extreme NVIDIA-stack optimization.

Summary #

  • Frameworks split into local execution (Ollama) and production throughput (vLLM, SGLang). Mixing the purposes misfires the choice.
  • vLLM and SGLang beat Ollama 3〜4x thanks to continuous batching and PagedAttention. The gap widens as users grow.
  • TGI entered maintenance mode in December 2025. Today’s production standard is vLLM, with SGLang as the alternative.
  • Quantization-format support, structured-output maturity, and new-model support speed are the selection axes beyond throughput.
  • Ollama for local, vLLM by default for multi-user, and compare SGLang for complex agent workloads is the practical baseline.
X