All posts
LLM App Operations #6: Security — Prompt Injection and Data Boundaries
Prompt injection is an attempt to change an app's behavior through input text, and in the era of RAG and agents it rides in through documents and tool results. We cover layered defenses instead of a single line, minimizing tool permissions, output validation, and the data boundaries of logging.
LLM App Operations #5: Reliability — Rate Limits, Retries, Fallbacks
429 and 529 are not outages, they are daily life. We build a structure that keeps running: how rate limits work (RPM and token limits), retries that respect retry-after, timeouts and streaming, and fallbacks for when nothing else works (model downgrade, queuing, graceful failure).
LLM App Operations #4: Batching — Half Price for Non-Urgent Work
Are you still sending work that does not need an immediate answer through the real-time API? The Batches API processes bulk requests asynchronously in exchange for a 50% discount on every token. We cover picking batch-worthy work, submitting and collecting, and operational patterns.
LLM App Operations #3: Prompt Caching in Practice
Caching the system prompt and tool definitions that repeat on every request cuts the input cost of that portion to one tenth. We cover the prefix-matching principle, stable prefix design, cache_control placement, and an audit for silent cache invalidation.
How Do Emergency Alerts Reach a Phone Without Knowing Its Number? Cell Broadcast
Emergency alerts look like text messages but are not: instead of addressing each phone by number, a cell tower broadcasts to every phone in range over a separate channel called cell broadcast. Why alerts reach millions in seconds and never queue behind congested networks, how alerts target a specific area, how severity tiers differ, and why your phone sometimes stays silent while everyone else's screams — explained for non-developers.
LLM App Operations #2: Cost — Token Accounting and Model Routing
The biggest lever for cutting cost is model choice. Measuring with count_tokens before sending, putting output on a diet, designing model routing by task difficulty, and tuning effort. We cover the order of operations for lowering cost while protecting quality.
LLM App Operations #1: Between Demo and Production — A Map of Operations
An LLM app that works and an LLM app you can operate are different things. We draw a map of operations along five axes — cost, latency, reliability, quality, and security — and build per-request instrumentation, the starting point for everything.
Advanced RAG #7: Capstone Project — Upgrading the Document Q&A Bot
Upgrade the internal document Q&A bot from LLM App Development Part 13 step by step with the techniques from this series. From measuring the baseline through swapping chunking, hybrid search, reranking, and citations, we watch how the metrics move at every step.
Advanced RAG #6: Building a RAG Evaluation Pipeline
We grow Part 1's baseline into a full evaluation system. Retrieval is scored with recall@k and MRR, generation with an LLM judge, and a single evaluation script that also measures hallucination rate runs as the regression test for every change.
Who Makes Emoji? Unicode, and Why They Look Different on Every Phone
New emoji are not added at any one company's whim — they are born through review by Unicode, the standards body that manages the world's writing. That an emoji is not a picture but a number like any letter, why the same emoji looks different on an iPhone and a Galaxy, how the family emoji is secretly several emoji glued together, how skin-tone variants work, and why some emoji show up as empty boxes on other phones — explained for non-developers.
Advanced RAG #5: Reducing Hallucinations with Citations
We tackle generation failures, where answers go wrong even though the right chunks were provided. We implement prompts that keep answers inside the evidence, the right to answer "I do not know," and per-sentence source attribution with the citations feature in Claude.
Advanced RAG #4: Query Transformation and Reranking
User questions are not good search queries. We reinforce both ends of retrieval: query rewriting that folds in conversation context, multi-query that asks from several angles, and reranking that precisely narrows a wide pool of candidates.