# When an Agent Fails, Should You Change the Model or the System?

> The useful distinction is not weak intelligence versus bad engineering. It is which intervention removes the failures that matter—and whether the gain survives realistic costs and checks.

Published: 2026-09-13 · Lumisonde (https://lumisonde.com/reports/agent-model-or-system/)
Citations JSON (per-claim receipts): https://lumisonde.com/reports/agent-model-or-system/citations.json
Continues: https://lumisonde.com/reports/harness-small-model-parity/

## The brief

**Question:** When an agent fails, should you replace its model, repair the surrounding system, or test both together?

**Stake:** The decision commits engineering effort and operating budget while the deployment still has to meet its requirements for completion, consequential errors, and response time.

**Answer:** Test a targeted system repair when you can name an obstacle in the agent's information, tools, workflow, or completion checks; test a model replacement when decision errors persist despite a usable route to completion. Treat those diagnoses as hypotheses, and distinguish an immediate model swap from a deployment adapted to the candidate. Compare the changes on tasks reserved from tuning, repeat trials, and test the combination when resources allow. Choose a configuration only if it meets the declared reliability and error constraints after counting failed attempts, human correction, integration, and maintenance—not just the price of a successful call.

In *Can a Harness Make a Small Model Match a Large One?*, we concluded that harness improvements matter, but whether a small model can substitute for a larger one depends on the task. The next practical question is how to decide, when an agent fails, whether to replace its model or repair the surrounding system.

## Introduction

When an agent performs poorly, test a model replacement when it repeatedly makes poor decisions despite having usable information and tools. Test a system repair when you can name an obstacle in what it sees, what it can do, or how completion is checked. Treat these as starting hypotheses, not diagnoses. A stronger model may cope with an awkward interface; a clearer interface may make difficult reasoning unnecessary.

Here, the model is the component interpreting context and selecting actions. The surrounding system supplies that context, exposes tools, manages state and retries, and checks results. Your deployed agent is their combination. The practical question is therefore which concrete change improves the failures you care about, under constraints you can afford—not which component deserves the blame.

Start with what the failure reveals, then identify which repair or replacement is worth testing. The final two steps are to compare those options fairly and decide which gains justify their full deployment costs.

## What does the failure actually tell you?

Imagine an agent asked to update a record and notify its owner. It updates the record, sends no notification, and reports that everything is complete. That establishes a mismatch between the requested and actual outcomes. It does not establish a reasoning failure: perhaps the notification tool was unavailable, its response was misunderstood, or the completion check inspected the final message instead of delivery status.

Completion language is particularly weak evidence: a study of tau2-bench and AppWorld documents agents reporting success while the environment records failure.[1] Start by checking the underlying result, including actions that should not have occurred. In the hypothetical example, inspect the record, notification delivery, recipient, and any duplicate sends—not just the reassuring closing sentence.

Next, inspect the information available immediately before the consequential decision, the action chosen, and the environment's response. Keep observations separate from explanations. “The agent used the wrong identifier” is an observation; “the model cannot reason about identifiers” is still a hypothesis.

Turn the leading hypotheses into small, discriminating tests. Supply a missing delivery-status observation while leaving the model unchanged. Repair a tool that rejects a valid action. If the required information was already available, replay the decision with a candidate model. Use a sandbox or restored test state so a diagnostic replay cannot notify someone twice. Record whether the intervention actually took effect before interpreting the outcome.

DoVer provides an empirical example of this approach: it edits suspected messages or plans and reruns the subsequent execution, while also finding that distinct interventions can repair the same failed task.[2] A successful repair therefore identifies a useful lever, not necessarily a unique original cause. Conversely, a failed repair may leave another obstacle in place. The goal is enough diagnosis to choose a testable change, not a perfect causal story before doing anything useful.

## When is a system repair the promising experiment?

Favor a targeted repair when its expected benefit is concrete: expose a missing status, distinguish an attempted action from a committed change, preserve a requirement across a long interaction, or make an error actionable. In the notification example, returning delivery state and requiring a completion check are different interventions. The first may help the agent finish; the second may only stop it from falsely claiming completion. Measure those outcomes separately.

System leverage can include spending less on an already adequate model. AgentDiet uses a separate reflection model to reduce accumulated trajectory content; in its Trae Agent experiments with Claude 4 Sonnet and Gemini 2.5 Pro across two coding benchmarks, it reports computational cost reductions of 21.1%–35.9% after reflection overhead.[3] That is evidence for a specific context-management intervention, not proof that shorter context always preserves the information your task needs.

Extra machinery can also make performance worse: in SWE-agent's historical GPT-4 Turbo experiments on 300 SWE-bench Lite tasks under a $4 per-instance cap, iterative search resolved 12.0% of tasks, versus 15.7% without the added search facility and 18.0% with summarized search.[4] This does not price a current deployment; it shows why “add a tool” is an incomplete proposal. What information does the tool expose, how much interaction does it require, and can this model use it?

For your first comparison, keep the model fixed and change one obstacle you can describe. Check whether the intended error category declines, then inspect cases that previously worked. A repair that suppresses missed notifications but introduces duplicate ones has exchanged failures, not solved the task. Prefer the smallest maintainable intervention that meets the requirement; complexity needs an observed benefit, not a presumption of sophistication.

## When is changing the model worth more?

A replacement deserves a test when the agent repeatedly confuses relevant conditions, selects an unsuitable action, or drops part of a compound request despite having a usable route to completion. You do not need to prove an absolute capability ceiling. You need a credible candidate and a way to see whether it reduces those residual errors on unfamiliar cases.

But choose the model together with its surrounding system. AgencyBench reports a reversal across ten selected scenarios: on the authors' scaffold (the surrounding agent framework), GPT-5.2 scored 57.4 and Claude-4.5-Opus scored 50.8; on Claude-Agent-SDK, GPT-5.2 scored 53.5 and Claude-4.5-Opus scored 71.3.[5] These are average rubric scores, with the benchmark using both executable checks and model judges, not probabilities of fully correct production completion.[5] The comparison establishes that the preferred model can depend on the surrounding framework; it does not isolate why the ranking changed or establish an equal-cost investment winner.

This makes two experiments worth distinguishing. An immediate swap keeps the current system as unchanged as compatibility permits: it asks what replacing the model buys now. An adapted deployment allows changes to prompts, tools, or context handling: it asks what the alternative product configuration can achieve. Both are useful, but only the first approaches isolation of a model change. Disclose even necessary compatibility edits, and count adaptation effort in the second comparison.

If the candidate helps only after a system repair, the combination may be the right investment. If a repair lets the current model satisfy the requirement, the replacement may be unnecessary. The published crossing of rankings is a reason to test those possibilities, not a recommendation to buy either named model.

## How do you compare the options fairly?

Use the following as a proposed local experiment, not a universally validated recipe. Begin with representative tasks and replayable starting states, including ordinary successes as well as the failures that prompted the investigation. Keep development cases for diagnosis and tuning separate from the final comparison. Held-out tasks, or holdouts, are the cases reserved for that final comparison and not used to tune the changes. AI Agents That Matter explains how inappropriate holdouts allow shortcuts that work only on benchmark tasks, while inconsistent evaluation can obscure whether improvements are genuine.[6]

Before tuning, define externally checkable completion, required permissions, unacceptable side effects, and the latency or spending limits. Audit disputed failures against the task specification; a defective evaluator should not decide which model you purchase. Do not let a candidate redefine success by producing a different closing message.

Compare four configurations when resources allow:

| Configuration | What it tests |
|---|---|
| Current model, current system | The baseline you must improve |
| Candidate model, current system | Immediate replacement value |
| Current model, targeted repair | Repair value without replacing the model |
| Candidate model, targeted repair | Whether the interventions reinforce or substitute for each other |

Use the same held-out tasks and outcome checks. Record model versions and settings, prompts, tool access, retry policy, context handling, resource caps, and actual resource use. A common cap does not guarantee equal consumption. If four configurations are unaffordable, compare the baseline and both single changes first; leave the combined option explicitly unresolved.

Repeat trials and retain failures, timeouts, and consequential violations. The original τ-bench distinguishes pass@k, success in at least one of k independent trials, from pass^k, success in all k trials averaged across tasks.[7] It also warns that its reward can pass an action performed without required user confirmation.[7] Accordingly, report ordinary completion, repeated-run consistency, and critical violations separately. Repetition cannot compensate for a success check that ignores the wrong behavior.

Compare changes on the same tasks, with uncertainty estimates appropriate to the sample, rather than treating a small aggregate difference as decisive. Inspect which failure categories changed. A repair may help one model more than another when judged by the same outcome measure; that difference does not prove a particular internal reasoning mechanism.

Set promotion and stopping rules before seeing the final scores. Promote only a configuration that meets the declared reliability and error constraints with enough evidence for the deployment's stakes. Reject a candidate that breaks a critical constraint even if its average score rises. If the comparison remains inconclusive, retain the current safe configuration, gather more cases, or narrow the supported task scope; do not turn uncertainty into a full rewrite by default.

## Which gains are worth paying for?

Decide whether you want the cheapest configuration meeting a reliability requirement or the best outcome within a fixed budget. Those are different optimization problems. In either case, first exclude configurations that violate hard constraints, then compare the feasible alternatives.

For each alternative, count inference and tool expense across all attempts, including failures and auxiliary calls; elapsed time; human review and correction; and integration and maintenance over an explicit volume and time horizon. Keep waiting time visible rather than hiding it inside a dollar estimate. For cost per acceptable completion, divide the total expense of the evaluated workload by its externally accepted completions—not just the expense of successful runs. An option with no accepted completions has no useful finite value on that measure.

Benchmark savings cover less than this full accounting: AgentDiet explicitly does not quantitatively compare latency, while AI Agents That Matter says human labor and maintenance were not extensively analyzed.[3, 6] Treat those unmeasured categories as inputs you must collect, not zeroes. A more expensive call could still be worthwhile if it avoids enough retries or human repair, but that arithmetic needs your measured rates.

After a candidate passes the held-out comparison, use a limited rollout with external checks and a rollback condition tied to the same critical errors. Revisit the choice when the task mix, system, model version, or prices change. The decision concerns a deployment configuration, not permanent allegiance to a model or architecture.

## Conclusion

Do not upgrade a model merely because an agent failed, and do not rewrite a system merely because system design matters. Identify a concrete obstacle or residual decision error, compare a replacement with a targeted repair, and test their combination when it could change the choice. Buy the improvement that survives held-out tasks, repeated execution, consequential-error checks, and full deployment costs. You need evidence that an intervention works—not certainty about which component deserves the blame.

## Limitations

These studies establish local intervention effects and model–system dependence, not a universal ordering of investments. Coding tests, simulated customer-service interactions, and rubric-scored long-horizon tasks do not share an interchangeable success measure. The dated SWE-agent result illustrates an interface failure mode, not the expected gain from changing a current agent. AgencyBench's ranking reversal does not supply matched adaptation costs or a unique causal explanation. Neither successful diagnostic replay nor failure after improved information proves an absolute model ceiling. The proposed comparison procedure must be sized and validated for the intended workload; no universal sample count, reliability threshold, or full-cost break-even point follows from these results.

## References

[1] From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents — https://arxiv.org/html/2606.09863
[2] DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems — https://huggingface.co/papers/2512.06749
[3] Reducing Cost of LLM Agents with Trajectory Reduction — https://arxiv.org/html/2509.23586v2
[4] SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering — https://arxiv.org/abs/2405.15793v3
[5] AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts — https://huggingface.co/papers/2601.11044
[6] AI Agents That Matter — https://huggingface.co/papers/2407.01502
[7] τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains — https://arxiv.org/pdf/2406.12045

---
Every [N] marker above is verifiable: fetch the citations JSON, match the marker id, and check the extraction (summary and key facts) plus placements against the source URL. Verification guide: https://lumisonde.com/for-agents/