RAG that survives production: grounding, evals & guardrails

A RAG pipeline has four moving parts: chunking, retrieval, generation, and verification. Most teams obsess over the model and ignore the other three. In production, the model is rarely the bottleneck — bad chunks and unmeasured drift are.

Chunk for meaning, not for tokens

Splitting documents into fixed 500-token windows is the single most common cause of wrong answers. It cuts tables in half and separates a claim from its caveat. Chunk on semantic boundaries — sections, list items, Q&A pairs — and attach metadata (source, date, section title) so the model can cite and the retriever can filter.

Retrieval is a ranking problem, not a search problem

• Use hybrid retrieval: combine dense vector similarity with keyword (BM25) so exact terms and acronyms aren't lost.
• Re-rank the top candidates with a cross-encoder before they reach the model — it's the cheapest accuracy you'll ever buy.
• Always retrieve more than you need, then filter by metadata (recency, permissions, source trust).

You cannot improve what you don't evaluate

Before launch, build a golden set of 100–300 real questions with verified answers. Score every change against it for faithfulness (is the answer supported by the retrieved context?) and relevance (did it answer what was asked?). Ship an eval, then ship the feature — never the other way around.

If your only test for a RAG system is 'it looked right when I tried it', you don't have a system — you have a slot machine.

Guardrails: refuse before you hallucinate

The safest answer is often 'I don't have enough information to answer that.' Add a grounding check that compares the generated claim against the retrieved passages and abstains when support is weak. Log every abstention — it's a map of exactly where your knowledge base has holes.