How to Make RAG Systems Abstain When Evidence Is Weak

Build abstention as a sequence of decisions. First ask whether the question is in scope for this knowledge base and user. Then retrieve and assess whether the returned material contains enough evidence. If it does not, try a bounded correction such as query decomposition, a lexical fallback, a permitted secondary corpus, or a clarification. Generate only from the surviving evidence. Finally, verify that each material claim is supported before releasing the answer.

An abstention is appropriate when the corpus lacks the answer, the relevant source is unavailable or forbidden, the evidence is too weak, the sources conflict without a resolution rule, or the question asks for more precision than the evidence provides. These are different states. A useful system does not collapse all of them into “I cannot help.”

Do not use the language model's confidence sentence as the only gate. Models can sound uncertain while correct and certain while unsupported. Do not treat a vector similarity score as a probability that the answer is true either. Calibrate several observable signals on questions from the actual corpus, and keep the threshold tied to the cost of a wrong answer.

Start with an explicit outcome model. “Answerable” means the retrieved evidence supports the whole requested answer. “Partially answerable” means it supports a bounded subset. “Conflicting” means credible sources disagree and the system cannot resolve the difference by date, authority, or scope. “Not found” means the search completed without sufficient evidence. “Outside scope” means the corpus was never meant to answer the question. “Access blocked” means relevant material may exist but this user cannot retrieve it.

These states change the response. Partial evidence should produce a narrower answer with the gap named. Conflicting evidence should preserve the disagreement. A permission boundary should not reveal the title or existence of a restricted file. An out-of-scope question can point to the corpus boundary, while a failed retrieval should expose that the system searched but did not find enough.

The distinction has a long history in question answering. SQuAD 2.0 added unanswerable questions that were deliberately relevant to the supplied paragraph and contained plausible answer-like material. This matters for RAG because irrelevant retrieval is often easy to reject; the harder case is a passage that looks close but omits the one fact needed.

Write acceptance criteria for each state before tuning prompts. Otherwise a team can improve the refusal rate by refusing almost everything, or improve answer coverage by guessing. Both can look good under a single metric.

A passage can be relevant without being sufficient. A cancellation clause may mention notice but omit the jurisdiction, effective date, or exception that determines the answer. Build a sufficiency gate that sees the question, the candidate evidence, and the expected answer shape. Ask which claims or slots must be supported, then mark which are present.

The Sufficient Context study separates failures caused by inadequate context from failures where a model misuses adequate context. Across the models and datasets it tested, stronger models often answered incorrectly instead of abstaining when context was insufficient. Its selective-generation experiment improved the fraction of correct responses among answered cases, but the result is not a universal threshold for another corpus.

Use retriever signals as inputs, not verdicts. Useful features may include top-result relevance, score margin, lexical coverage of identifiers, source authority, date validity, diversity across sources, and whether required metadata constraints were satisfied. Add an evidence-level classifier or rubric that asks whether the passages jointly answer the question. Store its rationale and cited spans for review.

Calibrate by question type. A policy number lookup may require an exact identifier match. A thematic synthesis may rely on several partially relevant sources. A threshold that works for one will mishandle the other. Segment evaluation by task, document family, language, and risk.

Weak first retrieval is not always an unanswerable question. Normalize identifiers, rewrite conversational references, decompose multi-part questions, expand an acronym, run lexical and semantic retrieval, or fetch the parent section around a precise match. Each correction should address an observed failure and remain within the same permission boundary.

Self-RAG trains a model to decide when to retrieve and to critique passage relevance, support, and generation quality through reflection tokens. Corrective RAG uses a retrieval evaluator to trigger different actions when retrieved documents appear correct, ambiguous, or incorrect. These papers show that retrieval quality can control the next step instead of feeding a fixed top-k into every answer.

Their architectures are not drop-in proof that self-critique is reliable. A production system can implement the principle more simply: one retry with a query rewrite, one hybrid fallback, and one clarification when intent remains ambiguous. Log which recovery path changed the evidence. Stop after a fixed budget so uncertainty does not become an invisible chain of increasingly broad searches.

External web search is only a valid recovery path when the product is allowed to leave its curated corpus and can label those sources clearly. In an internal policy assistant, silently replacing missing company evidence with public web text may be worse than abstaining.

Even sufficient passages do not guarantee a grounded answer. The generator can merge two sources incorrectly, add an unsupported qualifier, reverse a negation, or cite a passage that is related but not entailing. Run a claim-level support pass after drafting: split the answer into material claims, map each to source spans, and classify it as supported, contradicted, or not established.

The RAGTruth corpus was created because retrieval-augmented outputs can still contain claims unsupported by or contradictory to the supplied evidence. Its task settings and model outputs are research data, not a complete production specification, but the core distinction is valuable: having a citation nearby is different from the citation supporting the claim.

Remove unsupported decorative details rather than letting them force a full refusal. If the evidence supports the applicable notice period but not the reason it changed, answer the first part and mark the second as unavailable. If a central claim fails support, regenerate with only verified facts or abstain.

Keep verification independent enough to catch generator habits. A second prompt to the same model may repeat the same mistake. Where stakes justify it, combine deterministic checks for dates, numbers, and identifiers with a separate entailment or review model and human escalation.

A good abstention is specific without pretending certainty. State the part you could not establish, describe the evidence boundary, and offer the smallest next step. For example: “I found the current cancellation clause, but it does not identify the notice rule for Singapore. I need the jurisdiction-specific endorsement or product schedule to answer that part.”

Show what was checked when disclosure is safe: corpus, date range, source type, or query interpretation. Offer a clarification if the question has two plausible meanings. Suggest a missing document or field only when the system can name it from the workflow, not because the model invented what usually exists.

Do not apologize at length, repeat the question, or bury a partial answer under refusal language. Do not reveal inaccessible source metadata. Separate “not in the evidence” from “false.” The absence of a clause in retrieved documents does not prove that no clause exists.

Design the interface so a user can attach the missing source, revise scope, inspect retrieved evidence, or escalate to a person. Capture that action as feedback. An abstention that repeatedly asks for the same unavailable document is a product defect, not caution.

Build a test set with answerable and unanswerable questions from the same knowledge base. Include missing entities, false premises, unsupported time ranges, questions requiring restricted evidence, near-match passages, ambiguous references, contradictory versions, and requests that are only partly answerable. Mark the expected state and the exact evidence required for any accepted claim.

UAEval4RAG proposes six categories of unanswerable requests and reports that no tested component configuration was consistently best at both answering and rejecting across its knowledge bases. That is a warning against copying a refusal prompt or threshold without corpus-specific testing.

Report coverage and risk together. Coverage is the share of requests the system answers; selective risk is the error rate among those answers. Also track correct abstentions, unnecessary abstentions, partial-answer quality, permission-safe behavior, and whether recovery found evidence without weakening source policy. Plot performance across thresholds rather than presenting one operating point as natural.

Choose the operating point by consequence. A research discovery tool can tolerate more exploratory answers when uncertainty and sources are visible. A system used for claim decisions, legal obligations, or medical guidance needs a lower accepted risk and clearer human review. Recalibrate when the corpus, retriever, model, or question mix changes.

The goal is not maximum refusal. It is dependable selectivity: answer when the evidence path is strong, narrow the response when support is partial, and stop when the system would otherwise manufacture certainty. If your RAG workflow gives a polished answer where it should expose a gap, share the real question, retrieved evidence, and expected response at sahil@granveo.com.