How RAG Systems Should Handle Conflicting Evidence

A useful conflict workflow has four stages. Detect that two candidate claims cannot both be true under the same interpretation. Classify why they differ: version, scope, ambiguity, source error, legitimate disagreement, or conflict with the model's remembered knowledge. Apply a documented resolution rule where one exists. If the evidence remains genuinely unsettled, show the alternatives with their sources instead of manufacturing one answer.

Do this before drafting polished prose. Once a generator blends several passages into one paragraph, the boundary between claims becomes difficult to recover. Build a small evidence table first: claim, source, publication or effective date, applicable entity, jurisdiction, authority, and the passage that supports it. The final answer should be derived from that table and keep a citation beside every contested statement.

The language model can help identify contradictions, but it should not invent the policy for deciding which source wins. A regulator's current rule may supersede an old internal memo. Two clinical studies may deserve to coexist because they examined different populations. Three copied articles do not outweigh one primary filing. Resolution depends on the work, not on similarity scores or confident wording.

Start by asking whether the sources make claims about the same thing. A policy for one product may not contradict a rule for another. A quarterly figure may differ from an annual figure because the period changed. Two people or companies can share a name. Entity, unit, jurisdiction, population, and time must align before a difference becomes a contradiction.

Version conflict occurs when a later source revises or supersedes an earlier one. Both passages can be historically accurate, but only one may govern the current decision. Perspective conflict occurs when credible sources interpret the same evidence differently. Evidence-quality conflict pits a primary record against a summary, rumour, extraction error, or manipulated document. Parametric conflict appears when retrieved evidence disagrees with what the model learned during training.

The DRAGged into Conflicts study proposes a taxonomy and benchmark for several conflict types in realistic search-augmented settings. Its experiments found that models often handled conflicts poorly, while explicit reasoning prompts improved responses but left substantial room for improvement. That is a useful boundary: conflict-aware prompting helps, but the application still needs structured evidence and deterministic rules.

Noise is different again. An irrelevant passage may mention the same words without asserting a competing fact. Treating every retrieval mismatch as a conflict creates needless uncertainty. The detector should compare claims after relevance and entity resolution, not compare entire chunks merely because their embeddings are close.

Conflict handling begins at ingestion. Store a stable source identifier, source type, publisher or responsible agent, retrieval location, publication date, effective period, jurisdiction, version, and relationships such as revises, quotes, derives from, or supersedes. Keep the original passage beside any extracted claim or generated summary.

The W3C PROV model separates entities, activities, and agents, and represents derivation, revision, responsibility, and time. A RAG system does not need to implement every PROV term, but the distinction is valuable. A document version is an entity; the process that produced a summary is an activity; the publisher or reviewer is an agent. Without those links, a polished summary can look as authoritative as the source it transformed.

Do not overwrite an old document when a new version arrives. Mark the old version's validity and connect the revision. Historical questions may need the earlier source, while current questions need the effective one. Capture both publication time and the time the claim applies to; a document published today can describe a rule effective next month.

Also detect duplicated lineage. Ten websites may repeat one press release. Count that as one underlying source unless an outlet independently verified the claim. Corroboration means independent evidence, not repeated text with different URLs.

Split candidate evidence into testable claims with qualifiers. ‘The waiting period is 30 days’ is incomplete if the source actually says it applies to one plan, after a particular start date, with named exceptions. Preserve those qualifiers when normalising the claim. Removing them can create a contradiction that the documents never contained.

A practical detector can combine rules and models. Exact identifiers, numbers, dates, negation, and version markers support deterministic comparisons. Natural-language-inference or language models can flag semantic contradictions between relevant claims. A second pass should decide whether the pair is a direct conflict, different scope, ambiguous entity, or insufficient evidence.

Every automated label needs a trace back to the passages. Claim extraction and contradiction classification can themselves be wrong, especially with tables, OCR, legal exceptions, and domain language. High-impact or low-confidence cases should enter a review queue rather than disappear behind a single confidence score.

Keep the detector separate from the resolver. Detection asks whether evidence disagrees. Resolution applies domain policy. Combining both in one prompt encourages the model to choose a fluent winner before the system has established why one source should prevail.

Write source precedence as data that people can inspect. A current statutory or contractual source may outrank an internal note for a compliance question. A signed decision may outrank a meeting summary for a case history. A primary dataset may outrank a commentary for a measurement. These are examples, not universal hierarchies; each workflow needs owners who can define and approve its rules.

Apply scope before authority. The most prestigious source can still be irrelevant to the user's jurisdiction, product, date, or entity. Then consider supersession and recency. Newer is not automatically better: an amended policy may have a future effective date, and an old incident report can remain the correct evidence for what was known at the time.

When no policy selects a winner, preserve plurality. State that sources disagree, describe the difference precisely, attach each citation, and say what additional evidence or human decision would resolve it. Avoid phrases such as ‘sources suggest’ when one source says yes and another says no. That wording turns a disagreement into false consensus.

Astute RAG studies conflicts between retrieved information and model-internal knowledge and proposes iterative, source-aware consolidation. The research supports treating post-retrieval conflict as its own stage. In production, however, the model's remembered answer is not an unnamed tie-breaker. If internal knowledge matters, expose it as an unverified hypothesis and test it against admissible sources.

The answer composer should receive structured claims and resolution status, not a pile of undifferentiated chunks. For a resolved conflict, state the selected claim with its scope and effective date, cite the governing source, and mention the superseded source only when it helps explain the change. For an unresolved conflict, present the alternatives in parallel and avoid a recommendation unless the workflow authorises one.

Time-sensitive questions need a freshness check at query time. The FreshLLMs work introduced FreshQA for fast-changing knowledge and false premises, and found that evidence count and ordering influenced answer correctness. A system should therefore retrieve current authoritative material deliberately, not assume that adding more recent-looking passages will correct an old answer.

Ambiguous questions may have several valid answers. The Retrieval-Augmented Generation with Conflicting Evidence paper evaluates ambiguity, misinformation, and noise together and reports that these combined conditions remain challenging for RAG baselines. Ask a clarifying question when one missing qualifier would change the answer. If interaction is impossible, label the interpretations instead of silently choosing one.

Multi-agent debate can be useful for examining competing evidence, but more model calls do not create source authority. Use them to surface assumptions, missing qualifiers, and counterarguments. The final resolution still needs a traceable rule and cited evidence.

Build test cases from real disagreements: a superseded policy, two jurisdictions, an amended figure, a duplicated rumour, conflicting expert views, a false premise, and a source that contradicts the model's likely prior answer. Label the conflicting claims, their scopes, the permitted resolution rule, and the expected answer behaviour.

Measure conflict-detection recall, false conflict rate, correct classification, resolution accuracy, citation support, preservation of minority but valid views, and appropriate abstention or escalation. Include an update test: after a source changes, how long until retrieval, summaries, graph facts, caches, and answers reflect the new version without erasing historical truth?

The RAGTruth corpus shows that RAG outputs can still contain unsupported or contradictory claims and provides word-level annotations for studying them. An end-to-end score alone can miss this. Review answers claim by claim and record whether each statement follows the selected evidence or reintroduces a discarded claim.

Inspect failures through the whole path: ingestion, entity resolution, retrieval, claim extraction, conflict detection, resolution policy, and generation. A wrong answer may begin with a missing effective date rather than a weak model. Fix the earliest broken stage.

A trustworthy system does not promise to eliminate disagreement. It makes disagreement inspectable and prevents presentation quality from outrunning the evidence. If your knowledge workflow has two credible sources and no clear way to decide what governs, share the sources, decision, and missing rule at sahil@granveo.com.