RAG (Retrieval-Augmented Generation)

What is RAG?

RAG grounds an LLM’s answers in your data instead of only its training data. At query time the system retrieves relevant documents (typically via embeddings and a vector database), places them in the model’s context window, and the model generates an answer citing that material.

How it works

1. Ingest — chunk documents, embed each chunk, store vectors plus metadata.
2. Retrieve — embed the user query, find nearest chunks (often combined with keyword/BM25 hybrid search and a reranker).
3. Generate — pass retrieved chunks + query to the LLM with instructions to answer from the provided context.

When to use it — and when not

• Use RAG when answers must reflect private, frequently changing, or very large corpora: internal docs, support knowledge bases, legal/product catalogs.
• Skip classic RAG when the corpus fits in the context window (long-context models handle hundreds of pages directly), or when the knowledge is stable and stylistic — that’s fine-tuning territory.
• The 2026 shift: for agent systems, “agentic retrieval” — giving the model search tools it calls just-in-time — increasingly replaces one-shot vector lookup. Pipeline RAG still wins for high-volume, low-latency Q&A where cost per query matters.

Cost reality

A RAG query costs embedding + vector search (cheap, sub-cent) plus LLM generation over query and retrieved chunks. Retrieving 10 × 1,000-token chunks per query multiplies your input token bill ~10×. Rerankers cut that by letting you retrieve wide but pass only the best 3–5 chunks to the generator.

What people get wrong

• Chunking blindly. Fixed 500-token chunks split tables and code. Chunk on semantic boundaries and prepend document context to each chunk (contextual retrieval).
• Skipping evals. Retrieval quality, not the model, is usually why RAG answers are bad. Measure recall@k on a labeled set before touching prompts — see evals.
• Pure vector search. Hybrid (vector + keyword) with a reranker beats either alone on almost every benchmark.

Try it in code

A minimal RAG loop — embed, retrieve by cosine similarity, generate from retrieved context:

import numpy as np

def cosine(a, b):
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

# 1. Ingest: embed your chunks once (any embeddings API)
chunks = ["Refunds are processed in 5 days.", "Support hours are 9-5 CET."]
chunk_vecs = [embed(c) for c in chunks]          # embed() = your provider's API

# 2. Retrieve: top-k chunks for the query
def retrieve(query, k=2):
    qv = embed(query)
    scored = sorted(zip(chunks, chunk_vecs), key=lambda x: -cosine(qv, x[1]))
    return [c for c, _ in scored[:k]]

# 3. Generate: answer ONLY from retrieved context
context = "\n".join(retrieve("how long do refunds take?"))
answer = llm(f"Answer only from this context, else say 'not found'.\n"
             f"Context:\n{context}\n\nQuestion: how long do refunds take?")

That is the whole architecture. Everything else — chunking strategy, hybrid search, rerankers — is optimization of steps 1 and 2.