How to Evaluate a RAG System Without Fooling Yourself

To evaluate a retrieval-augmented generation system, test at least three layers separately: whether it retrieved the evidence needed for the question, whether the answer stayed faithful to that evidence, and whether the answer actually completed the user's task. Then add the behavior that ordinary benchmark questions miss: abstaining when the corpus has no answer, handling conflicting or outdated sources, respecting access boundaries, and staying useful within an acceptable latency and cost.

The same bad answer can have different causes. The right document may never enter the context. It may arrive, but the model may ignore it. Or the model may give a grounded answer that misses the task. One end-to-end score tells you something is wrong; a traceable evaluation tells you what to repair.

Start with a small set of real questions and reviewed evidence, not thousands of synthetic prompts. Record the expected source or passages, the essential facts an answer should cover, and cases where the correct behavior is to say that the available evidence is insufficient. Run this set on every material change to parsing, chunking, indexing, retrieval, reranking, prompting, or models. Automated metrics can widen coverage, but humans should calibrate what those metrics mean for the work.

RAG joins several components. The original retrieval-augmented generation paper combined a generator's parametric memory with passages fetched from a dense index. Production systems now add parsers, metadata filters, query rewriting, hybrid search, rerankers, citation formatting, and policy checks. Each extra component creates another place where a plausible answer can detach from the right evidence.

Write a failure taxonomy that reflects your workflow. For a research assistant, missing a dissenting paper may be a completeness failure. For a policy support tool, quoting a superseded clause is a freshness failure. For an internal workspace, retrieving a correct document from the wrong team's private collection is a permissions failure even if the final prose looks excellent.

Keep the categories observable. ‘Hallucination’ is too broad to diagnose. Split it into unsupported claims, contradictions with retrieved evidence, invented citations, and claims that may be true in general but are absent from the authorised corpus. The last case is especially important: a model's background knowledge can make an answer factually plausible while defeating the reason you added retrieval.

Collect questions from search logs, interviews, support cases, research tasks, and the decisions the system is meant to inform. Rewrite sensitive examples only enough to make them safe, preserving the structure that made them difficult. A balanced set should include direct lookup, synthesis across sources, multi-step relationships, comparison, chronology, and questions that cannot be answered from the corpus.

For each question, have a domain-aware reviewer identify the minimum evidence needed and draft a short scoring rubric. Avoid requiring one canonical paragraph when several answers could be valid. Instead, list essential claims, acceptable variants, prohibited claims, and the correct source version. Store stable document and passage identifiers so a retrieval change can be compared against the same evidence target.

Hold out part of the set. If every example is visible while prompts and rerankers are tuned, the evaluation becomes a development worksheet rather than evidence of generalisation. Refresh it when the corpus, user mix, or task changes.

The BEIR benchmark is a useful warning against narrow retrieval tests. Its authors evaluated different retrieval approaches across 18 datasets and found meaningful trade-offs across domains and methods. Your private benchmark can be much smaller, but it should preserve the variety that matters in your own collection rather than averaging unlike questions into one comfortable number.

Run the retriever against the reviewed evidence set without generation. Recall at k asks whether the required evidence appeared within the first k results. Precision at k asks how much of that limited context was relevant. Mean reciprocal rank rewards placing the first useful result early. Normalised discounted cumulative gain is helpful when passages have graded relevance rather than a binary label. None is universally best; choose the metric that matches how the generator consumes context.

Inspect results by question type, source format, department, language, and document age. An overall recall score can conceal that the system works on clean webpages but misses tables, or retrieves current policies while failing on historical comparisons. Also measure the context budget. Raising k may improve recall while flooding the generator with near-duplicates and distractors.

Do not evaluate embeddings in isolation and assume the RAG system improved. Test the complete retrieval path: parsing, chunk boundaries, metadata, lexical or semantic retrieval, filters, reranking, and deduplication. Keep a simple lexical baseline. BEIR found BM25 to be robust across its benchmark, while more expensive reranking and late-interaction approaches often performed strongly on average. The lesson is not that one method always wins; it is that a fashionable retriever still needs to beat a relevant baseline on your questions.

Once retrieval is visible, evaluate the generated answer claim by claim. Faithfulness asks whether the retrieved passages support what the answer says. Completeness asks whether the answer covered the essential supported points. Relevance asks whether it answered the user's actual question. Correctness may compare the response with a reviewed reference, but should allow wording and structure to vary. Citation quality adds two checks: the cited passage must support the adjacent claim, and important claims should not remain uncited.

Reference-free frameworks can accelerate this work. Ragas proposed metrics for retrieved-context focus, faithful use of context, and generation quality without requiring a human reference for every item. ARES evaluates context relevance, answer faithfulness, and answer relevance using trained judges, synthetic training data, and a small human-labelled set for statistical correction. These are useful patterns, not substitutes for defining what success means in your domain.

RAGChecker makes the diagnostic idea explicit by separating retrieval and generation metrics and evaluating claims at a finer level. That distinction prevents a strong generator from masking weak retrieval, or abundant retrieval from masking unsupported prose. Whatever framework you use, manually review a sample of passes and failures. Judge models can share biases with the system under test, miss domain nuance, or reward polished answers. Calibrate thresholds against human decisions and recheck them when the judge model changes.

A clean question with one relevant paragraph is the easy case. Add controlled stress tests. Remove the answer from the corpus and expect abstention. Insert topically similar passages that do not answer the question. Provide two sources that disagree. Add an older policy beside its replacement. Split the required evidence across documents. Include a false assumption in the question and check whether the system accepts it.

The RGB benchmark organises RAG behavior around noise robustness, negative rejection, information integration, and counterfactual robustness. Those categories translate well into practical slices. They reveal whether the generator can ignore distractors, refuse when evidence is absent, combine several pieces, and resist retrieved misinformation instead of treating every chunk as truth.

For a connected knowledge workspace, add relationship tests: can the system distinguish ‘supports’ from ‘mentions’, follow a decision to the evidence available at that time, and show when a later source superseded it? Graph retrieval may help multi-hop questions, but it introduces its own evaluation targets: entity resolution, edge accuracy, path relevance, and provenance. Judge the final answer and the route that produced it.

Run the fixed evaluation set before releasing a material change and compare results by slice, not only by average. Save the query, corpus version, retrieved passage IDs and scores, prompt and model versions, answer, citations, latency, and token or service cost. Without that trace, a regression becomes a screenshot and a debate.

Offline tests will still miss real behavior. In production, sample answers for domain review, watch queries with no strong retrieval result, and record when users reopen sources or reformulate the question. Treat these as signals, not perfect labels.

Set release gates around the failures that matter most. A small latency improvement should not justify worse abstention in a regulated workflow. Higher answer completeness should not come from filling gaps without evidence. And a metric improvement that does not change a human decision may not be worth the extra complexity or cost.

Small systems do not need an evaluation platform. A versioned table of fifty hard questions, reviewed passages, rubrics, outputs, and failure labels can be enough to begin. The important part is continuity: the same evidence trail should let you explain why one system version is better, where it remains weak, and what changed when a source was updated.

Granveo is exploring that traceable layer between sources and decisions. If you have a RAG workflow that looks good in demos but fails on a particular class of real questions, share the question pattern, evidence path, and failure at sahil@granveo.com. The broken trace is more useful than a generic request for better AI.