RAG pipeline checklist

This guide is for anyone building or reviewing a retrieval-augmented generation (RAG) system: a pipeline that fetches passages from your own documents and hands them to a language model so its answer is grounded in them. Most RAG failures are retrieval failures, and most retrieval failures are decided at ingestion. Work through the stages in order. The project at /projects/tiny-rag-local-embeddings/ builds a small one end to end, and /learn/beginner/08-embeddings-and-similarity/ covers the embedding side.

Two paths through a RAG system. The offline path turns documents into an index. The online path turns a question into a grounded answer, and every stage on it is a place to measure.

The checklist

Stage	Check	Default to start from	Why it matters
Ingestion	Every document has a stable id, source URL or path, title, last-modified date and access permissions	One metadata record per chunk, stored next to the vector	You cannot cite, refresh, delete or filter what you did not record
Ingestion	Text extraction is checked by eye on ten documents, including a table and a scanned page	Keep headings, drop boilerplate (navigation, footers), keep table rows as lines	Garbage in the index is retrieved with full confidence
Ingestion	Deduplication by content hash, and near-duplicates by embedding similarity above about 0.95	Keep the newest copy	Duplicates fill the top k with the same passage
Chunking	Chunks of 200 to 500 tokens with 10 to 20 % overlap, split on headings and paragraphs before fixed length	300 tokens, 50 overlap, heading text prepended to each chunk	Too small loses context, too large dilutes the embedding and the prompt budget
Chunking	Each chunk makes sense on its own	Prepend the document title and section path	The model sees the chunk, not the document
Embedding	Model chosen for the language and domain, with its max sequence length above your chunk size	A general-purpose open model such as all-MiniLM-L6-v2 (384 dims, 256-token limit) for English prototypes, a larger model for production	A chunk longer than the model’s limit is silently truncated
Embedding	The same model and normalisation for documents and queries, with asymmetric prefixes if the model wants them	Record the model name and version with the index	Mixed versions produce vectors that are not comparable
Index	Flat (exact) search up to about a million vectors, HNSW beyond that, IVF when memory is tight	Flat, with normalised vectors and a dot product	Approximate indexes lose recall for speed. Measure the loss
Index	Metadata filters (tenant, date, permission) applied inside the search, not after	Pre-filtering, or a post-filter with a larger k	Post-filtering a top 10 can return nothing
Hybrid	A BM25 (lexical) index alongside the vector index, merged by reciprocal rank fusion	Weight equally, retrieve 50 from each	Dense retrieval misses exact names, codes, version numbers and rare words
Reranking	A cross-encoder reranks the top 30 to 100 candidates down to the 3 to 8 that go in the prompt	A small open cross-encoder reranker	Bi-encoders rank by rough topic. The reranker reads query and passage together
Prompt	Passages numbered, with source and date, instructions before the passages, the question last	“Answer from the passages. Cite as [n]. If the passages do not contain the answer, say so.”	Citations make the output checkable and the failures visible
Prompt	Token budget enforced: passages trimmed to fit, never the instructions	Reserve a third of the context for the answer	Truncation from the end drops the question
Evaluation	A labelled set of at least 100 questions with the gold passage id and a reference answer	Build it from real queries, label with the people who own the documents	Without gold passages you cannot separate retrieval errors from generation errors
Evaluation	Retrieval recall at k and MRR on the gold passages, reported per change	Recall at 5 and recall at 20	If the gold passage is not in the top k, nothing downstream can fix it
Evaluation	Answer faithfulness (every claim supported by a cited passage) and answer correctness, judged by people on a sample and by a model on the rest	A rubric, a validated judge, confidence intervals	Fluent answers that cite the wrong passage are the common failure
Monitoring	Every query logged with the retrieved ids, scores, the prompt hash and the model version	A replayable record per answer	A complaint a week later is useless without it
Monitoring	Index freshness, embedding model version, empty-result rate, mean top score, latency per stage	Alert on empty results and on top-score drift	Drift shows up here before users report it
Monitoring	Re-ingest on document change, rebuild the whole index when the embedding model changes	Scheduled diff against the source	Stale passages cite documents that no longer say that

A flat cosine index in twelve lines

Flat search is a matrix product. With normalised vectors the dot product is the cosine similarity, and for anything under a million chunks it is fast enough and exact. Start here and move to HNSW only when you have measured the need.

import numpy as np
from sentence_transformers import SentenceTransformer

docs = [
    "The training loop calls zero_grad, backward and step in that order.",
    "Gradient clipping bounds the global norm before the optimiser step.",
    "A learning-rate scheduler is stepped once per epoch, after the optimiser.",
    "Mixed precision keeps weights in fp32 and runs matmuls in bf16.",
    "Chunk documents to 200 to 500 tokens with a small overlap.",
    "HNSW indexes trade memory for fast approximate nearest neighbours.",
    "BM25 scores exact lexical matches and complements dense retrieval.",
    "A reranker scores query and passage together and reorders the top 50.",
    "Recall at k measures whether the gold passage made it into the context.",
    "Faithfulness checks that every claim in the answer is supported by a passage.",
    "Log the retrieved ids with every answer so failures can be replayed.",
    "Rebuild the index when the embedding model changes, never mix versions.",
]
query = "how do I measure whether retrieval found the right passage"

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2", device="cpu")
index = model.encode(docs, normalize_embeddings=True)        # (12, 384), unit length
q = model.encode([query], normalize_embeddings=True)[0]       # (384,)

scores = index @ q                                            # cosine, because both are unit vectors
top = np.argsort(-scores)[:3]
for rank, i in enumerate(top, start=1):
    print(f"{rank}. {scores[i]:.3f}  {docs[i]}")

1. 0.479  BM25 scores exact lexical matches and complements dense retrieval.
2. 0.394  Recall at k measures whether the gold passage made it into the context.
3. 0.358  Log the retrieved ids with every answer so failures can be replayed.

The right passage (recall at k) is second, behind a sentence that shares the word “retrieval” but does not answer the question. This is the ordinary behaviour of a small bi-encoder, and it is the case for the reranker stage: a cross-encoder that reads the query and each candidate together would put the second sentence first. It is also why recall at 5, not top-1 accuracy, is the retrieval metric to tune first.

Chunking in more detail

Strategy	When	Trade-off
Fixed tokens with overlap	Unstructured prose, a quick first index	Cuts sentences and tables mid-way
Structural (headings, paragraphs, list items) then fixed-size fallback	Documentation, manuals, policies	Needs a decent parser per format
Sentence windows: embed single sentences, return the surrounding window	Precise question answering	Twice the bookkeeping, better precision
Parent-child: embed small chunks, return the parent section	Long technical sections	Larger prompts
Per-row or per-record	Tables, tickets, product catalogues	A row without its header is meaningless, so prepend it

Measure chunk size in the embedding model’s tokens, not characters. Check the distribution after chunking: a long tail of 5-token chunks (headings alone, page numbers) is a sign the splitter needs fixing.

Index choice

Index	Search	Memory	Build	Use when
Flat	Exact, linear scan	Vectors only	None	Under about a million vectors, or whenever you need a ground truth to measure the others against
HNSW	Approximate, graph walk, high recall at low latency	Vectors plus a graph, roughly 1.5 to 2 times the raw vectors	Minutes to hours, slow inserts at scale	Millions of vectors, read-heavy, memory available
IVF (with product quantisation)	Approximate, searches a few of many clusters	Much smaller with PQ	Needs a training pass on sample vectors	Tens of millions of vectors, memory-constrained, some recall loss accepted

Whatever the index, measure its recall against the flat index on your own queries before trusting it. A default HNSW setting that gives 0.99 recall on a benchmark may give 0.9 on your data.

The checklist

A flat cosine index in twelve lines

Chunking in more detail

Index choice

Sources