RAG vs Fine-Tuning vs Long Context: Getting Knowledge Into an LLM

Once an LLM adoption discussion gets anywhere, one question always arrives: “how do we get our company’s data into the model?” There are three options: RAG, which retrieves and injects; fine-tuning, which modifies the model’s weights; and long context, which puts the documents straight into the prompt. The conclusion up front: changing facts and document knowledge default to RAG; fine-tuning is for locking in output format, voice, and domain behavior; and if the entire knowledge base is small and stable, long context plus prompt caching is the simplest starting point. The three are not mutually exclusive — they combine. Cost figures are as of mid-2026.

Side-by-side comparison #

RAG: retrieve the knowledge and inject it per query #

RAG retrieves relevant document chunks for each incoming question and attaches them to the prompt. Its strengths are operational. Add a document and it affects answers immediately; you can cite which document backed the answer; and per-user access control resolves naturally at the retrieval layer. Even as the knowledge base grows past millions of tokens, the prompt stays at the few thousand retrieved tokens, so inference cost decouples from knowledge size.

The weaknesses are structural too. Answer quality is capped by retrieval quality. If the right document isn’t found, the best model still gets it wrong — diagnosing those failure modes is covered in RAG Deep Dive #1. The added moving parts — embeddings, chunking, a vector store — are a real cost as well. Building the pipeline from scratch is covered in the document Q&A bot project.

Fine-tuning: a tool for teaching behavior, not knowledge #

The most common misconception is “fine-tune on our data and the model will know it.” Injecting new facts via fine-tuning is inefficient, and it comes with side effects: existing capabilities can degrade, and the model tends to blend trained facts plausibly. What fine-tuning does well is fixing behavior: always answering in a specified JSON schema, holding a brand voice, raising accuracy on domain-specific classification and extraction, and folding a long recurring instruction block into the model to cut latency and token costs.

Costs have come down. Instead of retraining all weights, the standard is training small adapters with LoRA or QLoRA — a single run on an open-weight model lands in the hundreds of dollars, and managed API fine-tuning bills per training token (tens of dollars per million). The real burden is operational, not financial: you need a training dataset and an evaluation setup, every knowledge update means retraining, and if you self-host, the serving decisions follow (quantization, GPU procurement).

Long context: put it all in and cache it #

With context windows now in the hundreds of thousands to a million tokens, the third option became practical. For knowledge the size of an internal policy handbook, a product manual, or API docs, you can skip retrieval and put the entire document set into the prompt. Add prompt caching and repeated calls get steep discounts on the document portion of input tokens plus lower first-token latency — making “just put it all in before building RAG” a reasonable opening move.

The boundary conditions are clear. The whole knowledge base must fit comfortably in context (roughly a few hundred thousand tokens or less), it must change rarely (to avoid cache invalidation), and the call pattern must repeatedly reference the same documents for caching to pay off. If the documents keep growing, token costs rise in direct proportion, and the recall of information buried mid-context degrades as documents get larger. That is the moment to move to RAG.

Combining them: a layering, not a contest #

Real deployments are mostly combinations. The base placement: facts via RAG, behavior via fine-tuning, small stable knowledge via context. Use RAG (not fine-tuning) for document Q&A; if that RAG’s answer format keeps drifting, add a format-fixing fine-tune; and keep the rules and glossary shared by every call in a cached system prompt. Whether to start on an API or self-host follows the same math as API vs self-hosting — and a fine-tuned model is a variable that tilts the scale toward self-hosting.

Selection order #

1. First separate knowledge from behavior: “it needs to know X” points to RAG or long context; “it needs to answer like X” points to fine-tuning. This one split settles half the debate.
2. Measure knowledge size and update frequency: under a few hundred thousand tokens with rare updates, start with long context plus caching. Larger, or updated daily, means RAG.
3. Check citation and access-control requirements: if answers must cite sources or respect per-user permissions, a retrieval layer is required — RAG, settled.
4. Add fine-tuning last: start only after confirming prompt instructions cannot hold the format or voice; otherwise the dataset investment goes to waste.
5. Sanity-check cost against the call pattern: compare monthly cost of RAG versus long context as call volume × average prompt tokens, and compute when fine-tuning’s prompt-shortening effect pays back the training cost.

Summary #

• Changing facts and documents: RAG. Fixed format, voice, behavior: fine-tuning. Small stable knowledge: long context plus caching.
• Using fine-tuning as a knowledge-injection tool is the most common mistake. It teaches behavior, not facts.
• Long context is the simplest start when knowledge stays under a few hundred thousand tokens and rarely changes. When documents grow, move to RAG.
• RAG’s ceiling is retrieval quality, not the model. When answers are wrong, diagnose retrieval first.
• Production setups combine all three: facts in RAG, behavior in a fine-tune, shared rules in a cached context.