When Sources Disagree: A Decision Model for Deep Research

Introduction

The central problem in Deep Research is not how to fit more text into a context window. It is how to prevent materials with different jobs from being mistaken for one another. A system rule constrains behavior. A user file may supply a private fact or merely a proposition to check. Model knowledge can suggest useful hypotheses but may be stale and is usually hard to trace. A retrieved document or live page can be inspected, yet can still be outdated, copied, or wrong. A tool result is an observation produced under particular inputs and execution conditions.

There is therefore no defensible universal ordering such as “live web over documents over model memory.” The better mental model is a claim–evidence–decision pipeline. The system first expresses candidate answers as scoped claims; then gathers evidence that supports, contradicts, or qualifies each claim; then records a decision to accept, condition, juxtapose, defer, or abstain. Source identity remains important, but as an input to adjudication—not as the adjudication itself.

The path through the design is to assign each input a role, normalize it through the composition loop, distinguish real conflicts from scope differences, adjudicate by claim-specific evidence fit, route remaining uncertainty to an action, and retain the provenance needed to review or reopen the decision.

What exactly is being combined?

Before combining content, separate the control plane from the evidence plane.

The control plane answers: What may the system do, and whose instructions govern? System and developer rules set behavioral and safety boundaries; the user specifies the task and may add constraints. Text encountered inside files, webpages, and tool output is normally data, not a newly authorized command. Indirect prompt-injection research identifies failure to distinguish informational context from actionable instructions as a core vulnerability.[5] An instruction hierarchy is therefore useful for deciding which commands to follow.[6]

The evidence plane answers a different question: What should the report say about the world? Command priority cannot establish factual truth. A high-authority instruction can define an assumption for the task, but it cannot make an exchange rate, experimental result, or legal provision correct. Conversely, a low-authority webpage may contain the best available primary record while remaining unable to alter the system’s instructions.

Within those two planes, the six inputs in the question have distinct default roles:

These are defaults, not a closed taxonomy. A user file can be the authoritative internal ledger for one task and an unverified rumor in another. A model’s remembered definition can be adequate for query expansion yet inadequate for a claim that readers must audit. The role belongs on the individual item and claim, not only on its channel of arrival.

How should material move from context to conclusion?

A practical composition loop is:

1. Frame the task. Record the question, requested time horizon, jurisdiction or population, acceptable source constraints, output requirements, and consequences of error.
2. Plan and acquire. Generate queries and tool actions; search public or private sources; open candidate documents; locate the passages or records that bear on the question.
3. Normalize into claims. Extract candidate claims with their subject, predicate, value, time, geography or population, conditions, and source location. Preserve enough surrounding context to interpret each claim.
4. Test coverage and conflict. Link supporting and refuting evidence, search for counterevidence to a depth set by claim risk, credible-source availability, and a bounded retrieval budget, identify source dependencies, and mark unanswered subquestions.
5. Revise. Change the plan when evidence exposes a new entity, date, ambiguity, contradiction, or information gap. More retrieval is an action selected for a reason, not a default response to every doubt.
6. Decide and synthesize. Promote only claims that pass the required checks; express conditions or disagreements where they remain; attach citations to the exact claims they support.

“Retrieved” should mark a workflow state, not confer evidentiary status. Product documentation shows users constraining sources, revising plans, and receiving cited reports after iterative search and tool use.[1, 2] Controlled research systems likewise distinguish opening a source and locating its relevant passage from merely receiving a search snippet.[3] Together, these sources support the observable loop, not a claim that vendors share the same internal architecture.

For implementation, three record families covering four machine-readable record types are a useful minimum:

This is a recommended engineering contract, not an industry standard. A candidate item should become citable evidence only when its underlying passage or execution can be inspected, its source and transformations are recorded, its scope is explicit, and its support relation can be checked. A claim should become publishable only when the required coverage is present, material conflicts are either resolved or visibly retained, and every source-dependent assertion has adequate claim-level support. If an upstream source changes, a tool rerun fails, or scope changes, dependent claims should be downgraded and queued for re-evaluation. Treat these states as an engineering recommendation and boundary, not as a validated universal promotion schema.

When are two sources actually in conflict?

Different strings are not necessarily incompatible facts. “Revenue was X in 2024” and “revenue was Y in 2025” may describe an update. A global estimate and a Taiwan-specific estimate may both be correct. Two dates associated with the same organization may denote founding, incorporation, or first production.

Conflict detection should operate on comparable claims, not whole documents. For each pair, ask:

1. Do the claims concern the same entity or event?
2. Do they assert the same property?
3. Do their time, geography, population, jurisdiction, method, and other conditions overlap?
4. If the scopes overlap, can both values still be true?

Only a “no” to the last question after the earlier alignments establishes a factual conflict. Work on search-augmented language models distinguishes genuine conflict from complementary information and shows that reading fuller context can dissolve apparent contradictions.[4] A separate design concern is that an underspecified claim can be verified against the wrong proposition. The field does not yet establish one sufficient set of claim fields for every Deep Research domain, so schemas should allow domain-specific extensions.

Once a candidate conflict is found, classify its cause before resolving it:

• Entity or event mismatch: names or dates were aligned incorrectly.
• Scope difference: both claims may hold for different periods, populations, jurisdictions, definitions, or thresholds.
• Version or state update: later evidence may invalidate an earlier state without explicitly negating it.
• Method disagreement: datasets, measurement procedures, definitions, or metrics differ.
• Substantive disagreement: comparable sources reach incompatible factual or interpretive conclusions.
• Execution variance: a tool rerun or stochastic research run returns a different observation.
• Adversarial instruction: source content tries to change agent behavior; this belongs on the control plane, not in the factual vote.

Explicit conflict representation matters because language models do not reliably perform all of this triage implicitly; on the controlled CONFLICTS benchmark for search-augmented systems, taxonomy-aware prompting improved compliance with the expected behaviors, while significant errors remained.[4] Counterevidence search should be the default control, with its depth and stopping rule set by claim risk, credible-source availability, and a bounded retrieval budget; “none found” must remain a valid outcome. Absence of an accessible rebuttal is not proof that the initial claim is true.

If claims truly conflict, what should the system trust?

Use evidence fit, not a fixed source leaderboard. Evaluate each claim–evidence relationship along independent dimensions:

• Task and scope fit: Does it answer this question for the relevant time, population, market, jurisdiction, and definition?
• Directness: Is it an original record or observation, or a summary of someone else’s claim?
• Publishing responsibility: Is the publisher accountable for creating or maintaining this particular information?
• Temporal and version fit: Was it valid for the period asked about, and is it superseded?
• Method transparency: Are the dataset, definitions, sampling, calculations, and limitations inspectable?
• Reproducibility: Can the observation or calculation be rerun under recorded inputs?
• Claim binding: Does the cited passage actually support the precise proposition, rather than merely discuss the topic?
• Independence: Do apparently multiple sources derive from distinct records, or repeat one upstream account?
• Counterevidence: What credible material would falsify or materially narrow the claim?

These dimensions should not be collapsed into universal weights. In controlled context–memory conflicts spanning tasks with different knowledge demands, models’ relative reliance on retrieved context and parametric memory changed with task requirements.[8] Direct Deep Research experiments also show that polished, authoritative-looking false material can penetrate multi-stage workflows.[7] Recency, institutional identity, and directness can all guide verification effort; none is sufficient by itself.

Count provenance clusters, not URLs. In a tightly controlled English evaluation of 13 open-weight models using synthetic source identities, repeating information from less credible sources could reverse observed source preferences.[9] When copying or a shared upstream record has been established, three pages repeating one press release form one evidentiary lineage, not three independent votes. The decision record should preserve the winning claim, credible alternatives, accepted and rejected evidence, the reason for the outcome, and the evidence that would reopen it.

When no candidate dominates on the relevant dimensions, the correct result is not forced consensus. The system can narrow the claim, present competing versions with dates and methods, state a conditional answer, or abstain. Adjudication is complete when the remaining disagreement is represented honestly and routed appropriately—not only when one side has been selected.

How should uncertainty change the next action?

One confidence number is too lossy. It can hide whether the model is uncertain, the evidence is insufficient, retrieval is stuck, sources conflict, or a required premise is missing. Research on RAG separates context sufficiency from the model’s ability to use sufficient context,[10] while retrieval-state research shows that repeated answers can agree because every run is conditioned on the same defective retrieval state.[11] Fluent consistency is therefore not a reliable substitute for evidence diagnostics.

Preserve at least three families of state:

• Answer state: stability across formulations or runs, sensitivity to evidence removal, and unsupported inference.
• Evidence state: direct, partial, irrelevant, missing, stale, dependent, conflicting, or method-limited support.
• Acquisition state: query coverage, inaccessible sources, repeated empty or homogeneous retrieval, tool error, and rerun variance.

Add task state where needed: an unconfirmed user premise, unresolved entity identity, missing private data, or a high-risk decision context. Then route from cause and risk rather than tone:

This routing matrix is a design skeleton, not a universal threshold policy. In production, counterevidence depth, thresholds, and budgets should be chosen on held-out tasks from the relevant domain, model, toolchain, and cost regime. More retrieval can introduce irrelevant evidence, while overly aggressive abstention can discard correct answers; the operating point must reflect both error cost and coverage.[10] For high-impact decisions, human review should receive the disputed claim, evidence, provenance, and unresolved question—not merely a confidence score.

Because every routed action should append to or update the decision record, the next question is what provenance makes that record reviewable.

What provenance makes the decision reviewable?

A references list is necessary for readers but insufficient for engineers. Evaluation work on Deep Research treats report quality and citation accuracy as separable dimensions,[12] and claim-level interfaces can expose unsupported assertions and omissions that ordinary source lists hide.[13] A linked sentence can still overstate, misread, or only partially use its source.

Maintain two connected views:

1. A support view links every externally checkable claim to exact evidence units and labels the relation: support, refute, quote, compression, calculation, or inference.
2. A process view records source versions and capture times; retrieval and tool actions; inputs, parameters, outputs, and errors; transformations; conflict and decision records; and downstream dependencies.

A reviewable system connects prompts, responses, decisions, tool interactions, and downstream outcomes.[14] At the claim level, it can link answer sentences to tool turns, evidence units, and semantic support relations.[15] These adjacent designs show useful connections; they do not prove that any one schema is sufficient for Deep Research.

Keep updates append-only where practical: a new source version or invalidation event should not erase the old node. Dependency edges can then identify which claims and report passages require rechecking. That enables local repair rather than either trusting a stale report or rerunning the entire task.

Auditability also has a privacy and cost boundary. Store enough structured history to reconstruct support and decisions, but do not assume that every token, private file excerpt, credential-bearing tool parameter, or hidden reasoning artifact should be retained or exposed. Apply minimization, access controls, retention rules, and role-specific views; increase trace detail at high-risk nodes.[16]

Conclusion

A robust Deep Research system does not decide that one source class always wins. It keeps the control plane and evidence plane separate, converts inputs into scoped claims, promotes only inspectable support, detects conflict after aligning scope, adjudicates by claim-specific evidence fit and source independence, and carries unresolved uncertainty into an explicit action or qualified answer.

The practical unit of reviewability is the decision record: what the system concluded, which evidence supports and challenges it, why alternatives were rejected or retained, what remains unknown, and what future evidence would reopen the case. That makes an answer not merely cited, but inspectable and correctable.

Limitations

Public product descriptions reveal a shared high-level pattern—planning, source selection, iterative acquisition, tool use, and cited synthesis—but not a common internal representation, source weighting scheme, promotion state machine, or production gate.[1, 2] The record types and routing policy above are therefore bounded engineering recommendations assembled from direct Deep Research evidence and adjacent RAG and provenance research.

The available evidence also does not show that these controls, by themselves, guarantee end-to-end factual accuracy, calibrated confidence, safety, or deployment reliability. Those require separate validation on representative long-form tasks, including source and domain drift, adversarial content, tool failures, latency and cost, population differences where relevant, human escalation behavior, and claim-level attribution. Promotion should depend on those evaluations; the conceptual architecture is not statistical assurance.