Tokens and Context Windows: The Units That Decide LLM Cost and Limits

LLM price sheets read “dollars per million tokens,” and model spec tables read “context window: 1M tokens.” If you can read those two numbers precisely, half of your LLM cost estimation is already done. The conclusion up front: a token is the smallest unit in which a model processes text, and the context window is the ceiling on the total number of tokens a single request can involve. Cost, performance limits, and architecture choices are all decided on top of these units. The broader API cost picture is covered in LLM API vs Self-Hosting; this post covers the units that calculation is built on.

Tokens: subwords, not words or characters #

Models process text neither as words nor as characters but as tokens. A vocabulary of frequently occurring string fragments is built from training data, and input text is split into those fragments. BPE (Byte Pair Encoding) style algorithms are the common approach.

• In English, frequent words often map to a single token, which works out to roughly one token per four characters.
• A long word like internationalization gets split into a few pieces, such as international + ization.
• Languages like Korean and Japanese occupy a smaller share of the vocabulary than English, so the same content produces more tokens. It varies by model, but budgeting 1.5〜2x the English count is a safe assumption.

One important fact follows from this. Token counts differ per model — strictly speaking, per tokenizer. The same document yields different counts on vendor A’s model and vendor B’s model, and there have been real cases where a tokenizer swap between model generations changed the token count of identical text by close to 30%. Never trust an estimate produced by another model’s tokenizer library; measure with the token counting API each vendor provides, against the model you will actually use.

Pricing: input and output have different rates #

LLM APIs bill input tokens and output tokens separately, and output costs 3〜5x more than input. Generation has to produce tokens one at a time sequentially, which costs more compute. So the same million tokens costs very differently between “read a long document and summarize briefly” and “generate a long article from a short instruction.”

For conversational products there is one more factor. LLM APIs are stateless: every turn resends the entire prior conversation as input. The input for turn ten contains all nine previous questions and answers. Per-turn input tokens grow cumulatively as the conversation continues, so total cost grows faster than linearly in the number of turns. Left unmanaged, long conversations quietly inflate the bill.

Two mechanisms mitigate this structure.

• Prompt caching: if the repeated leading portion (system prompt, conversation history) is cached server-side, the cached span is processed at around one-tenth of the list price. Cache writes carry a small premium, but using the same prefix twice or more comes out ahead. Note the cache is a strict prefix match from the beginning — putting an ever-changing value like a timestamp near the front of the prompt invalidates the whole cache.
• Batch API: for work that does not need real-time responses, most vendors offer a 50% discount on batched requests.

The context window: how much the model sees at once #

The context window is the total token budget a single request can use. The system prompt, conversation history, attached documents, tool definitions, and the model’s own output all have to fit inside it. As of 2026, 200K〜1M tokens is standard for major models, and some offer more. One million tokens is roughly ten novels’ worth of text.

Two numbers need to be kept apart.

The max_tokens parameter you set per request bounds the output. Set it too low and responses get truncated mid-sentence, and re-requesting a truncated response doubles your cost. Whether a response was cut off is reported in the stop reason field of the response.

The trap of large context: fitting it in is not the same as using it well #

Growing context windows made “just paste the whole document” viable, but there are two traps.

1. Performance degradation: with very long context, models have been observed to miss information located in the middle (the lost-in-the-middle effect). Recent models have improved a lot, but do not assume accuracy with a maxed-out context equals accuracy with a short one. Placing important information near the beginning or end remains a valid technique.
2. Cost: input tokens are billed on every request. Paste a 1M-token document and ask ten questions, and you pay the input cost ten times. Caching softens this, but RAG — retrieving only the relevant parts — is often structurally cheaper.

Whether to paste the whole document, retrieve parts of it, or train it into the model is covered in detail in RAG vs Fine-Tuning vs Long Context.

A practical checklist #

• Measure first. Run representative prompts through the vendor’s token counting API and multiply by projected traffic for a monthly estimate. Do not reuse numbers measured against a different model.
• Budget for token inflation in non-English content. Multiply English-based benchmark costs by 1.5〜2x for languages like Korean and Japanese.
• Manage history. Build summarization or truncation of old turns in from the start. Some vendors offer server-side automatic compaction.
• Design prompts for caching. Fixed content (system prompt, tool definitions) goes first; per-request content goes last.
• Watch max_tokens together with the stop reason. If truncation goes undetected, incomplete responses reach users as-is.

Summary #

• Tokens are subword units, and counts differ per tokenizer. Measuring with the counting API of the model you will actually use is the only accurate method.
• The same content uses more tokens in languages like Korean and Japanese than in English. Factor that into cost estimates.
• Input and output have different rates, and conversations get expensive fast because history is resent every turn. Prompt caching and the batch API are the main mitigations.
• The context window caps input plus output combined; max output tokens is a separate spec.
• Being able to fit everything in does not make it the best option. Long context has performance and cost traps, and the choice against RAG depends on the workload.