Building RAG Systems That Actually Work

The Basic RAG Architecture

A RAG system has two phases: indexing and inference. During indexing, documents are chunked, converted to vector embeddings, and stored in a vector database. During inference, the user’s query is also converted to an embedding, used to retrieve the most relevant document chunks, and those chunks are inserted into the LLM’s context window along with the original query. The LLM then generates an answer grounded in the retrieved content.

Common Failure Modes

Poor Retrieval Quality

If the retrieval step doesn’t return the right documents, the LLM cannot generate a good answer regardless of how capable it is. Poor retrieval is usually caused by suboptimal chunking (chunks that are too large, too small, or split at semantic boundaries), weak embedding models, or missing metadata filtering. Investing in retrieval quality almost always yields better results than tweaking the generation prompt.

Context Window Overload

Inserting too many retrieved chunks into the context window can cause the LLM to lose focus or truncate information. Techniques like re-ranking (using a more sophisticated model to re-order retrieved chunks by relevance), compression (summarizing retrieved chunks before insertion), and selective retrieval (only retrieving what is needed for the specific query) help manage context effectively.

Hallucination Despite Retrieval

LLMs can still hallucinate even when relevant documents are in context, especially if the documents contain ambiguous information or if the model is not properly instructed to ground its answer in the provided sources. Explicit instructions (“Answer based only on the provided documents. If the answer is not in the documents, say ‘I don’t know’.”) and citation requirements (“Cite the specific document section that supports each claim”) significantly reduce hallucinations.

Advanced RAG Techniques

Hybrid Search

Combining dense vector retrieval (semantic search) with sparse retrieval (keyword-based, like BM25) consistently outperforms either approach alone. Hybrid search retrieves using both methods and merges results, capturing both semantic matches and exact keyword matches.

Query Expansion and Rewrite

User queries are often underspecified or use different terminology than the knowledge base. Query expansion rewrites the user’s query to be more specific, adds synonyms, or generates multiple query variants—all of which improve retrieval recall.

Evaluation Frameworks

You cannot improve what you cannot measure. RAG evaluation requires assessing both retrieval quality (precision@k, recall@k, MRR) and generation quality (faithfulness to retrieved docs, answer relevance, citation accuracy). Frameworks like RAGAS, TruLens, and custom evaluation pipelines are essential for systematic improvement.

Production Considerations

RAG systems in production need monitoring for retrieval failures, answer quality degradation, and latency. They need versioning for both the knowledge base and the retrieval/generation pipeline. And they need guardrails: what to do when retrieval returns nothing, when the LLM refuses to answer, or when the user asks an out-of-scope question.