# How an AI Agent Should Decide Which Decisions Deserve Credit or Blame

> A final score does not explain a trajectory. Reliable improvement reconstructs the decision, tests fair alternatives, measures uncertainty, and applies the smallest reversible repair.

Published: 2026-09-20 · Lumisonde (https://lumisonde.com/reports/agent-trajectory-credit-assignment/)
Citations JSON (per-claim receipts): https://lumisonde.com/reports/agent-trajectory-credit-assignment/citations.json

## The brief

**Question:** When an AI agent succeeds or fails, how should it decide which earlier decisions deserve credit or blame—and what should change as a result?

**Stake:** A final score describes the outcome of an entire trajectory, while an update must act on some particular part of the system. That gap exists whether the run succeeds or fails.

**Answer:** Treat the outcome as evidence, not an explanation. Reconstruct the decision and what the agent knew at the time; compare only alternatives it could actually have chosen; then replay or approximate those alternatives and estimate how much they change the outcome, with uncertainty left visible. Separate plan errors from execution errors, and do not force delayed or interacting effects into a precise blame score when the evidence cannot support one. Turn the diagnosis into the smallest reversible change, test it against the incumbent, and abstain or collect more data when replay fidelity, coverage, or downside risk is inadequate.

An AI agent should not learn directly from “success” or “failure.” It should ask which decision changed the probability of the result, given what was knowable at the time, and then test the smallest repair that follows from that diagnosis. Otherwise a lucky success can reinforce a reckless process, while a sound decision followed by bad luck can be punished.

The useful output is not a story about why one run felt good or bad. It is an uncertain, decision-level estimate: *if this choice had been replaced by another choice that was genuinely available, how would the distribution of outcomes have changed?* Turning that estimate into an update requires five steps:

1. Reconstruct the decision point.
2. Define admissible alternatives.
3. Run—or honestly approximate—the counterfactual.
4. Estimate the effect and preserve its uncertainty.
5. Apply the smallest reversible repair.

Each step can fail independently. A rich trace can still omit the real choice; a plausible alternative can still be unfair in hindsight; repeated replay can still simulate the wrong world; and a correct diagnosis can still produce an overbroad repair. The method works by making those assumptions visible rather than compressing them into a single blame score.

## 1. Reconstruct the decision point

Before assigning credit, reconstruct what the agent observed, what it could control, and where it committed to a course of action. A useful record is ordered and versioned. It contains the task instruction, the observations available before each decision, the option class and selected action, tool arguments and results, external-state changes, intermediate checks, and the terminal outcome. It should also identify the model, prompt or policy, tools, evaluator, instrumentation schema, and relevant request settings. OpenTelemetry's GenAI conventions provide concrete fields for agent and conversation identity, model settings, messages, tool calls, retrieval, and separately linked evaluation records, although the vocabulary was still moving between specifications in the captured version.[1] Agent-learning work likewise formalizes a trajectory as an instruction followed by action–observation history, step, transition, and outcome.[2]

That is a practical core, not a proven minimum. If the system generated candidate actions, record them; if replay controls or seeds exist, record them. But a trace should say what was observed, what changed, and which components produced each event—not pretend to contain every hidden cause.

The attribution unit should normally be a **commitment-bearing decision point**, not every token and not merely the later event where damage becomes visible. Candidate boundaries include adopting a plan, choosing a subgoal, accepting a source, invoking a tool, handing work to another component, granting permission, or stopping. Log both plan transitions and environment-facing actions, but treat the boundary itself as a hypothesis: the consequential unit might be the plan, the action, or their combination.

Intermediate judgments must remain separate from the events they judge. Store the evaluator's identity and version, target, score or label, explanation if supplied, and available calibration information. Never overwrite an action with “bad step” as though the label were an observation. Visible reasoning is not a complete causal record either: across six prompted-hint settings, one study found that reasoning traces often disclosed a used hint in fewer than 20% of affected examples.[3] A rationale may help locate a decision worth testing; it cannot certify why the agent chose it.

## 2. Define admissible alternatives

A decision can receive credit or blame only relative to an alternative. The alternative must have been available under the original information state. Replacing a choice with an action that depends on facts revealed later is a hindsight rewrite, not a counterfactual test.

For each candidate decision, specify a small set of alternatives before running the comparison. An admissible alternative should:

- be defined precisely enough to execute or simulate;
- respect the permissions, tools, and operating constraints present at the time;
- use no observation that arrived later;
- change the decision under test without silently changing several others; and
- have at least some support under the original policy or option class.

This is where trace analysis, critics, learned value estimates, attention, temporal proximity, and model-generated explanations are useful: they can rank candidate decisions and suggest alternatives. They are not causal verdicts. On the Who&When benchmark, the best evaluated method identified decisive failure steps with only 14.2% accuracy, even though locating the responsible agent was easier.[5] They are useful for ranking what to test; without an intervention or defensible identification design, they do not establish what caused the outcome.

## 3. Run—or approximate—the counterfactual

For a candidate decision at time *t*, preserve the history available just before *t*, substitute a plausible alternative, and compare the distribution of later outcomes. The target quantity is an outcome contrast such as

\[
\Delta_t(a') = \mathbb{E}(Y \mid do(A_t=a')) - \mathbb{E}(Y \mid do(A_t=a)),
\]

where *a* is the observed choice, *a′* is an admissible alternative, and *Y* is a predeclared outcome measure. Agent-specific work operationalizes this idea by intervening on a step and running the trajectory forward repeatedly.[4]

In plain language: recreate the same decision context, replace the observed choice with another allowed choice, and compare what happens afterward.

Full replay is not always safe or credible. Real tools create side effects; providers may be nondeterministic; changed contexts can alter every later opportunity. Causal Agent Replay calls this a run-forward confound: once two runs reach different contexts, it becomes difficult to hold their later randomness comparable. Its demonstrations use mocked reproducible tools and exclude real side-effecting tools.[4]

Choose the strongest design the setting permits:

1. Use matched repeated replay when the environment and tools can be reproduced faithfully.
2. Use a sandbox or simulator when live replay would create unacceptable side effects.
3. Use matched historical cases or controlled probes when only parts of the trajectory can be recreated.
4. Use structured human review when intervention is impossible—and label the result as weaker evidence rather than causal proof.

In every case, predeclare the decision, alternatives, outcome measure, and what will be held fixed. Inspect whether the intervention changed later opportunities, not only later actions. If replay fidelity fails, the result should become unresolved rather than be converted into a confident blame score.

## 4. Estimate the effect—and preserve uncertainty

One favorable branch is an anecdote. Run enough matched or repeated continuations to estimate an effect and its uncertainty. Repeated rollouts and confidence intervals measure how much results vary across samples. They do not remove bias from a bad intervention, evaluator error, an inaccurate simulator, hidden confounding, or downstream contexts that have diverged. Delayed credit and causal credit are not the same operation.

Then separate three problems that are easy to collapse:

- **Delay:** a later reward may depend on an earlier decision. Methods such as RUDDER redistribute terminal return toward earlier predictive events,[6] while TRACE assigns turn-level signal from later information gains in tool-using agents.[7] These methods can make sparse feedback easier to learn from, but prediction of return is not proof of causal responsibility.
- **Hierarchy:** a poor tool call may be the execution of an already flawed plan. A better analysis first asks whether the high-level subgoal was sound, then whether execution within that subgoal was sound. HiPER implements this separation with persistent subgoals, switching decisions, and two-timescale advantage estimation.[8]
- **Interaction:** two decisions may be safe alone but harmful together. Single-decision removal can miss synergy or double-count blame. Shapley-style allocation asks how much each choice contributes across different combinations, but testing every combination is usually infeasible; sampling only some combinations adds further uncertainty.[9]

Most importantly, the available evidence does **not** validate Shapley-style allocation for synergistic decisions inside one long tool-agent trajectory. The closest evidence concerns coalitions across agents, not interactions among one agent's sequential choices. Use targeted pairwise tests where credible replay exists; escalate to sampled coalition analysis only when its assumptions are defensible; and leave residual uncertainty unassigned rather than force every outcome into a neat sum of blame.

The result of this step should therefore be one of three things: a material estimated effect with uncertainty appropriate to the consequence; an imprecise signal that justifies collecting targeted data; or an unresolved attribution that should not drive learning yet.

## 5. Apply the smallest reversible repair

Attribution narrows the repair; it does not dictate it. Convert the suspected causal locus into a small repair specification: the diagnosed behavior, the desired replacement, forbidden behaviors, expected tool use, and the outcome that would falsify the repair. REFLECT demonstrates this pattern by turning a localized failure into a structured repair plan and using targeted replay near a rollback point to test whether the correction flips the result.[10]

Choose the nearest controllable surface, but treat the mapping as a hypothesis:

| Diagnosed problem | First surface to test | Do not jump straight to |
|---|---|---|
| Missing or stale information | retrieval, memory, or observation policy | model-weight training |
| Poor options or premature commitment | planner, decomposition, or search budget | a broader evaluator |
| Bad choice among adequate options | prompt, selection policy, or targeted training | more retrieval |
| Mis-scored good and bad behavior | evaluator, grader, or reward definition | punishing the actor |
| Unsafe capability exposure | permissions, confirmation, sandbox, or connectivity | relying on a better score |

No matched study establishes this as a universally optimal mapping. Its value is experimental discipline: alter one plausible cause at a time, so a better outcome can teach you something.

Start with the smallest reversible candidate and compare it with the incumbent on repeated trials. Test the motivating failure slice, nearby variants, ordinary capability tasks, regression tasks that should retain prior performance, and risk-specific cases. Agent-evaluation guidance distinguishes capability evaluations from high-retention regression evaluations and stresses repeated-trial success rates.[11] High-impact changes should add sandboxing, human approval, shadow execution, restricted permissions, or rollback rather than rely on a mean score alone; a deployed coding-agent example ties safety training and network controls to specific observed risks and intended use.[12]

Coverage and uncertainty determine the decision. Adjacent safe-policy work suggests preserving baseline behavior in situations where the data contain few examples of the relevant state and action[13] and adopting a candidate only when a confidence-aware lower bound clears the incumbent.[14] These are design analogies for tool-using agents, not transferable formulas. There is no universal numeric update size, confidence threshold, rollout count, or traffic percentage supported across prompts, planners, safeguards, and model parameters.

The operating rule is simple:

- **Promote narrowly** when the estimated effect is material, uncertainty is acceptable for the consequence, and failure-slice and regression tests clear predeclared gates.
- **Collect targeted data** when the repair is plausible but imprecise or outside logged coverage.
- **Keep the incumbent or add a safeguard** when replay is not credible, regressions appear, or downside severity is high.
- **Expand gradually** only after the narrow change survives those checks; every expansion is a new experiment.

## Conclusion

Success and failure are evidence, not explanations. A reliable agent-learning loop reconstructs a commitment-bearing decision, defines alternatives that were actually available, tests them through the strongest credible counterfactual design, estimates effect without discarding uncertainty, and changes only the nearest controllable surface through a reversible experiment.

The standard is not “find the step to blame.” It is to produce a causal estimate whose assumptions are visible and a repair that can itself be falsified. Sometimes that process will identify a decision worth changing. Sometimes it will show that the evidence is too weak to learn from the episode. Both are better outcomes than confidently teaching the agent the wrong lesson.

## Limitations

Observable traces are necessary for reconstruction but are not proven sufficient for replay or causal attribution. Agent-specific counterfactual methods still rely heavily on mocked tools, constrained benchmarks, outcome evaluators, or predictive models; real side effects and divergent downstream contexts weaken identification. Intermediate judges and visible rationales are fallible, and confidence intervals cover sampling error rather than structural bias.

The largest open gap is interaction credit within a single tool-agent trajectory. The evidence also contains no matched comparison showing which repair surface is best for the same attributed failures and no universal numeric rule translating effect size, uncertainty, coverage, and risk into an update magnitude. Deployment thresholds must therefore be calibrated to the application, consequence, and evaluator rather than copied from adjacent reinforcement-learning regimes or vendor examples.

## References

[1] OpenTelemetry, “Gen AI semantic convention attributes.” — https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai
[2] Wang et al., “STeCa: Step-level Trajectory Calibration for LLM Agent Learning.” — https://huggingface.co/papers/2502.14276
[3] Anthropic et al., “Reasoning Models Don't Always Say What They Think.” — https://huggingface.co/papers/2505.05410
[4] “Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failures.” — https://arxiv.org/html/2606.08275v1
[5] Zhang et al., “Which Agent Causes Task Failures and When? On Automated Failure Attribution in LLM Multi-Agent Systems.” — https://icml.cc/virtual/2025/poster/45823
[6] Arjona-Medina et al., “RUDDER: Return Decomposition for Delayed Rewards.” — https://proceedings.neurips.cc/paper/2019/hash/16105fb9cc614fc29e1bda00dab60d41-Abstract.html
[7] “TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents.” — https://arxiv.org/html/2607.13988v1
[8] “HiPER: Hierarchical Plan–Execute RL for Multi-Turn LLM Agents.” — https://icml.cc/virtual/2026/poster/64058
[9] Pollmann and Staudacher, “On Least Squares Approximations of Shapley Values and Their Sampling.” — https://www.mdpi.com/2673-9321/6/2/18
[10] “REFLECT: Intervention-Supported Error Attribution for Silent Failures in LLM Agent Traces.” — https://arxiv.org/html/2606.09071v1
[11] Anthropic, “Demystifying evals for AI agents.” — https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
[12] OpenAI, “Addendum to GPT-5.2 System Card: GPT-5.2-Codex.” — https://deploymentsafety.openai.com/gpt-5-2-codex
[13] Laroche et al., “Safe Policy Improvement with Baseline Bootstrapping.” — https://ewrl.wordpress.com/wp-content/uploads/2018/09/ewrl_14_2018_paper_3.pdf
[14] Chandak et al., “Towards Safe Policy Improvement for Non-Stationary MDPs.” — https://all.cs.umass.edu/pubs/2021/Chandak%20et%20al%20-%20Towards%20Safe%20Policy%20Improvement%20for%20Non-Stationary%20MDPs.pdf

---
Every [N] marker above is verifiable: fetch the citations JSON, match the marker id, and check the extraction (summary and key facts) plus placements against the source URL. Verification guide: https://lumisonde.com/for-agents/