LLM Quantization Compared: FP16, INT8, INT4 and GGUF, AWQ, GPTQ

The first tool for fitting the same model onto a smaller GPU is quantization. It stores weights at lower precision to cut VRAM in exchange for a little accuracy — and “how much you give up for how much you save” varies by method. The conclusion up front: GPU multi-user serving goes to AWQ, local execution to GGUF, and 4-bit (INT4) is enough in most cases — but keep INT8 or FP16 when reasoning and code are central. VRAM sizing is covered in LLM Server Sizing; this post is the choice of how to shrink that VRAM.

Precision: VRAM scales with bit count #

The axis of quantization is how many bits you store weights in. The original is usually 16 bits (FP16/BF16), and halving the bits roughly halves VRAM.

Reading it is simple. Drop to INT4 and VRAM is a quarter of FP16, so a 70B model fits comfortably on a single H100 (80GB); FP16 needs two. Actual VRAM adds the KV cache on top of weights, so add headroom to the table — but this proportional relationship is the skeleton of the choice. This saving is what lowers the GPU cost in the break-even math.

The three formats: same 4-bit, different uses #

Up to INT4, it is a story of “how much you shrink.” The practical choice is “in which format.” The three formats diverge by use.

• GGUF: the format used by llama.cpp and Ollama. It has fine-grained levels from Q2_K to Q8_0 and supports CPU+GPU hybrid execution, making it strong in VRAM-constrained local environments. Q5_K_M and Q6_K are near the original while Q2_K degrades noticeably, so locally the balance point is usually Q4_K_M〜Q5_K_M.
• AWQ (Activation-aware Weight Quantization): quantizes while protecting the top ~1% of weights deemed important by activation magnitude. It retains about 95〜97% of FP16 quality — the best accuracy of the three — and offers high throughput in GPU serving. It is the default for vLLM multi-user serving.
• GPTQ: a GPU quantization method that saw wide use before AWQ, retaining about 90〜96% of FP16. It benefits from abundant tooling and pre-quantized models, but there are regions where it trails AWQ on accuracy and speed.

Where INT4 degrades quality #

“Mostly held” at 4-bit does not mean always safe. Some tasks show pronounced degradation.

• Math, code generation, reasoning-heavy tasks: where a single token’s error makes the whole result wrong, INT4’s slight quality drop surfaces as actual mistakes. These workloads are safer on INT8 or FP16.
• Simple chat, summarization, classification: where a small quality drop is imperceptible, dropping to INT4 to save VRAM and cost is the win.

That is, the workload’s nature sets the precision floor. Do not read only benchmark numbers (perplexity, accuracy); first compare before and after on your real task and confirm it is acceptable. There are results showing AWQ beats GPTQ on reasoning and instruction-following even at INT4, but either way 4-bit goes through validation on reasoning tasks.

Matching the serving framework #

Format choice is bound to the serving framework choice. Pick them separately and they misalign.

• Local execution with Ollama/llama.cpp → GGUF. Other formats are unsupported or limited.
• GPU serving with vLLM/SGLang → AWQ or FP8 win on throughput. GPTQ works too, but AWQ is the safe default for new deployments.
• FP8: an 8-bit floating point hardware-accelerated on recent GPUs (Hopper and later), with less quality loss than INT8 and high serving throughput — expanding its place in production.

The selection order #

1. Set the precision floor first: INT8 or higher if reasoning, math, and code are central; start at INT4 for chat, summarization, and classification.
2. Narrow the format by environment: GGUF for local, AWQ (or FP8) for GPU serving.
3. Fit VRAM: pick the lowest precision at which the model, plus KV cache, fits the target GPU. Read it alongside server sizing.
4. Validate on the real task: compare before and after on real inputs, not benchmarks, and confirm the quality drop is within tolerance.
5. For throughput, use a serving-optimized format: for multi-user, AWQ and FP8 outthroughput GGUF.

Summary #

• Quantization cuts VRAM in proportion to bit count. INT4 is a quarter of FP16, taking a 70B model from 140GB to 35〜40GB.
• Formats diverge by use: GGUF for local, AWQ for GPU serving, GPTQ the prior generation. AWQ has the best accuracy at 95〜97% of FP16.
• INT4 degrades noticeably on math, code, and reasoning tasks. Keep those on INT8 or FP16.
• Choose the format bound to the serving framework: Ollama takes GGUF, vLLM takes AWQ and FP8.
• Finish the method choice with quality validation on the real task, not benchmarks.