# When an AI agent’s past becomes useful

> An agent learns from experience only when past outcomes change later choices through scoped, testable, revisable guidance that survives variation without hidden regressions or disproportionate cost.

Published: 2026-09-19 · Lumisonde (https://lumisonde.com/reports/agent-learning-from-experience/)
Citations JSON (per-claim receipts): https://lumisonde.com/reports/agent-learning-from-experience/citations.json
Continues: https://lumisonde.com/reports/agent-task-completion/

## The brief

**Question:** When does an AI agent’s retained experience become reusable learning rather than stored history?

**Stake:** A growing memory can preserve useful evidence, but it can also amplify accidental success, faulty feedback, and bad analogies. The practical issue is whether stored experience improves later work without eroding earlier abilities or consuming more resources than the improvement is worth.

**Answer:** Past outcomes become learning when they change later decisions through a bounded loop: record what happened, separate observations from explanations, form a conditional lesson, test it, apply it only when its conditions match, and revise its influence after use. Episodes, distilled guidance, executable skills, selection policies, and parameter updates play different roles; none is universally best. Genuine improvement must appear on relevant unseen work, alongside tests for harmful transfer, retention, and total learning and inference cost. Existing studies support parts of this loop in specific settings, not a universal architecture or an unrestricted promise of lifelong improvement.

In *When can an AI agent say the task is complete?*, we argued that an agent should claim completion only when inspectable evidence supports the requested result and any material gaps remain explicit. That makes one task's outcome more trustworthy. For a persistent agent, the next question is what that outcome should change: when does a success or failure become reusable learning rather than stored history?

An agent can save every prompt, tool call, and result and still repeat the same mistakes. Storage makes the past available; it does not decide what the past means, when it applies, or whether following it caused a better outcome. Experience becomes learning only through a closed loop: preserve what happened, form a provisional lesson, test that lesson on later work, select it only when its conditions match, and revise or retire its influence when outcomes disagree.

That loop also explains the difference between remembering an experience and learning a reusable method. A remembered episode says, “On this task, these actions produced these observations and this result.” A method says, “When these relevant conditions hold, take these actions, expect these intermediate signals, and stop or recover in these ways.” The method is more compact and easier to execute across cases, but it is also a stronger claim. It may generalize—or it may turn one lucky or misleading episode into a durable mistake.

## What changes when experience becomes learning?

The first change is not a new memory format. It is a better account of the event.

Suppose an agent is asked to compile a complete report. It retrieves one page of records, writes a polished summary, and misses later pages. “The report failed” is only a verdict. A useful experience record separates at least five things:

1. the task and constraints;
2. the actions the agent actually took;
3. the observations and tool responses it received;
4. the check or feedback that established the outcome; and
5. the proposed explanation, marked as a hypothesis rather than a fact.

This separation matters because the same visible failure could come from different causes: the agent never noticed pagination, the continuation token was malformed, the source timed out, or the request did not require completeness after all. A repair should target the observed break, not merely the negative label.

Checks improve attribution, but they do not make it infallible. In Reflexion’s coding experiments, agents used generated tests and verbal feedback to guide later attempts without updating model weights; the paper also documents the central danger: incorrect code can pass an inadequate self-generated test, while correct code can be rejected by a faulty one [1]. Other work shows that a structured, task-specific verification procedure can make self-correction useful in some settings, but self-correction can still make an answer worse, and the evaluated tasks remain narrow [2]. The lesson is not “reflection works” or “self-critique fails.” It is that the agent must preserve what the check actually establishes and avoid promoting the check’s output into a broader causal story.

Success deserves the same treatment. A completed task does not validate every action in its trajectory: some steps may have been unnecessary, a guess may have happened to work, and the evaluator may have ignored a defect. Conversely, a failed trajectory may contain a correct substep worth preserving. Learning therefore assigns credit below the level of the whole episode. It asks which decision was supported by the subsequent observation, which decision remains uncertain, and which alternative should be tested next.

For the report, the candidate lesson is not “make more requests.” It is: “When completeness is required, inspect the response for a continuation mechanism; follow it until the source signals exhaustion; then verify the count or terminal condition.” That formulation names a trigger, a procedure, and evidence of completion. It is already more useful than the failure record—but it is still only a candidate method.

## Memory, guidance, and methods are different layers

That candidate method is not limited to one form or place of adaptation. An agent can retrieve an earlier trajectory as an example, insert a distilled instruction into its context, call a saved program, update a memory’s usefulness score, or change the model’s parameters. These mechanisms should not be collapsed into “the agent learned,” because they create different capabilities and failure modes.

Episodic memory preserves a particular event in detail. It is valuable when the new task closely resembles the old one, when rare exceptions matter, or when the agent needs provenance for a later diagnosis. Its weaknesses are volume and accidental detail: retrieving too much can crowd the context, and superficial similarity can surface a case whose assumptions do not match.

Extracted guidance compresses one or more episodes into a statement such as a warning, heuristic, or checklist item. ExpeL, for example, combines retrieved successful trajectories with natural-language insights derived from successes and failures, then lets later task attempts consume both [3]. Compression can expose the common structure, but it can also remove the boundary that made a lesson safe.

An executable skill packages a behavior so the agent can invoke and compose it. Voyager stores programs in a Minecraft skill library after an iterative generate–execute–check process and demonstrates reuse in a new world [4]. That is stronger operational reuse than a note in a log, but it remains evidence about a particular environment, API, curriculum, and verifier. Code can encode a general procedure; it can also hard-code one answer.

Finally, the learned object can be the selection policy rather than the content. A system may retain the same episodes while updating which ones are useful after observed rewards. That changes later behavior without rewriting the memory or retraining the base model [8]. Parameter updates are another possible locus, but they are not required for persistent behavioral adaptation.

These layers are complementary, not a ladder with episodes at the bottom and procedures at the top. Direct comparisons in procedural-memory research show that distilled procedures can improve average success or shorten execution while raw trajectories remain better on particular models, tasks, or constraints [5]. ExpeL likewise reports that retrieved examples and extracted insights can contribute differently across environments [3]. The relevant question is therefore not “Which representation is smartest?” but “Which representation supplies the right information, at an acceptable cost, for this decision?”

A robust design often keeps two linked records: the original episode as evidence and a compact candidate method as an editable claim. The method points back to its supporting and contradicting episodes. This lets the agent act efficiently without losing the detail needed to challenge its own summary.

## A cycle for learning without hardening mistakes

The steps below are a bounded synthesis of components supported across different systems, not a universally validated end-to-end architecture.

### 1. Capture an outcome-bearing episode

Record task conditions, relevant state, actions, observations, resource use, and the outcome check. Separate direct observations from interpretations. Preserve successful substeps from a failed attempt and suspicious steps from a successful one instead of assigning the terminal score to the entire history.

### 2. Propose the smallest conditional lesson

Compare a failure with a later success on the same task, or compare several successful cases, to identify the decision that changed. ExpeL operationalizes this with candidate insights that can be added, edited, up-voted, or down-voted [3]. The procedure is useful as an extraction mechanism, but voting by the same model is evidence of internal agreement, not independent proof that the lesson caused success.

Write the lesson so it includes:

- an applicability test: when should it be considered?
- a procedure: what should the agent do?
- expected signals: what should happen along the way?
- a failure test: what would show that it does not apply or did not work?
- provenance and version: which episodes produced it, and how has it changed?

### 3. Test locally before broad promotion

Run the candidate against the actual environment or verifier before storing it as trusted guidance. Voyager’s loop executes generated programs, incorporates environment messages and interpreter errors, and stores a skill only after a critic accepts completion; after repeated unsuccessful generations, it moves to another task [4]. This local execution check is required before the skill is stored as trusted guidance; it is not certification that the skill will work under changed inputs.

Broader promotion needs a stronger comparison. Evaluate the agent with the candidate method, with the closest raw episode, and with neither on new cases that share the relevant structure. Include misleadingly similar cases where the method should abstain. If the candidate improves only the original task, keep it as a case-specific repair. If it transfers but fails for a recognizable subgroup, narrow its conditions. If it harms performance or adds more cost than value, reduce its influence or reject it.

There is no evidence-backed universal threshold for how many successes are enough. The promotion rule must be set for the consequences of the application: a reversible formatting shortcut can tolerate weaker evidence than an action that changes production data.

### 4. Select by applicability and observed usefulness

Retrieval should produce candidates, not commands. Before use, compare the current task’s constraints, tools, environment, and required outputs with the method’s scope. PATH-Bench implements an instructive three-way choice over retrieved material: retain it when it is compatible, summarize it when only part transfers, and suppress it when it has no concrete reusable connection or risks misleading the agent [6]. Its decision is temporary rather than a permanent deletion, which is an important safety property.

Selection can also combine semantic relevance with an outcome-derived utility estimate. MemRL first filters memories by similarity, then ranks them using usefulness values updated from observed rewards [8]. Similarity alone can retrieve a plausible but harmful analogy; utility alone can retrieve a historically successful memory that is irrelevant now. The two signals answer different questions: “Does this fit?” and “Has this helped?”

### 5. Revise influence after use

After applying a method, record the new outcome and update the method’s scope, content, or selection weight. Agentic Context Engineering uses identifiers to target particular playbook entries and helpful/harmful counters to update those entries rather than repeatedly rewriting the entire context [7]. This preserves detail that a wholesale summary could erase. Utility-based systems can leave the text unchanged and revise only its future selection weight [8].

Conflicts should remain visible until evidence resolves them. Two opposing lessons may both be valid under different conditions; merging them too early produces a vague rule, while deleting the less frequent one may erase a rare but critical exception. Retirement should therefore usually mean “do not surface by default,” with the underlying episode and version history retained for audit or reactivation.

This cycle is deliberately conservative: capture, hypothesize, test, conditionally promote, select, observe, and revise. Its purpose is not to make memory grow monotonically. It is to make the influence of memory earn its place repeatedly.

## Proving later work actually improved

A corrected retry shows that the agent can use feedback on the same task. It does not show that the agent has acquired a method for future tasks. The evaluation must separate at least four effects:

- **same-task correction:** another attempt at the original problem;
- **structured transfer:** a new task reuses an earlier subproblem, workflow, or constraint;
- **held-out generalization:** a new task was not used to build the memory;
- **retention:** earlier capabilities remain available after more experience accumulates.

AgentCL makes these distinctions explicit with controlled task streams, frozen-memory reuse, and held-out tasks; it finds that memory can help when earlier structure is deliberately reusable, while naive and held-out settings can show limited gains or degradation [9]. PATH-Bench uses recurring evaluation probes that do not update memory while intervening tasks do, exposing gains that later erode and interference from accumulated experience [6]. Together, these results make a single end-of-run average inadequate.

A fair test compares the same base agent under several conditions: no accumulated experience, retrieved raw episodes, extracted methods, and—when relevant—both together. Give each condition the same evaluation tasks and report the resources each actually consumes. Include related tasks where transfer should help, unrelated tasks where memory should be neutral, misleadingly similar tasks where reuse should be refused, and recurring probes for old capabilities. Keep test episodes out of the memory-building set.

Measure more than final accuracy. At minimum, track task success, harmful transfer, retention on recurring probes, attempts, tool calls, tokens, latency, and the cost of extracting, validating, retrieving, and revising lessons. A shared maximum budget is not the same as equal spending: one system may consume far more of the cap. Report learning cost separately from later solving cost so readers can see whether an expensive lesson is eventually amortized.

Freeze the evaluation protocol before inspecting the candidate’s results. Define what counts as a related, unrelated, or misleadingly similar task. Specify when retrieved material is offered, selected, summarized, or suppressed, and check the evaluator on representative successes and failures. Choose the number of stochastic repeats and how they will be aggregated; account for context allowance as well as tokens, tools, latency, and learning cost; and set application-specific promotion and unacceptable-harm criteria. These are local design commitments, not universal thresholds. Report them with the result so a favorable outcome cannot be created afterward by relabeling tasks, changing selection behavior, or moving the pass line.

For this framework, a practical stopping rule is to promote a method only when it improves relevant unseen work, does not create unacceptable harm on confusable or earlier tasks, and repays its added complexity at the expected reuse rate. Narrow, suppress, or retire it when those conditions fail. If the evidence supports only same-task correction, call it correction—not reusable learning.

## Conclusion

Experience-based learning is not a larger archive and not a ritual of reflection after every attempt. It is a controlled change in how later decisions are made. Remembering supplies detailed evidence and useful analogies. A reusable method is a conditional claim about how to act across relevant cases. Because that claim can be wrong, it needs provenance, applicability checks, outcome-based selection, and revision.

The decisive test is prospective: does the agent perform better on genuinely later work, while refusing bad analogies, retaining earlier capabilities, and paying a defensible cost? If yes, the past has become useful learning. If not, the system has accumulated history—or, at best, learned how to retry one task.

## Limitations

The evidence supports components of this cycle, not an unrestricted promise of lifelong improvement. The cited systems span coding, question answering, tool use, Minecraft, and constructed task streams; their verifiers, APIs, models, memory budgets, and definitions of transfer differ. Several procedures validate task completion or update usefulness estimates without establishing the causal truth of each extracted lesson. No universal promotion threshold, conflict-resolution rule, retirement policy, or compute break-even point has been demonstrated. Most of the results discussed here concern external memories or skills rather than parameter updates, so they do not establish what the base model itself has learned [3, 4, 6, 8, 9]. A production system should treat the cycle above as an evidence-bounded design and evaluation method, then calibrate its checks and stopping rules to its own domain and consequences.

## References

[1] Reflexion: Language Agents with Verbal Reinforcement Learning — http://arxiv.org/abs/2303.11366v4
[2] Large Language Models Can Self-Correct with Key Condition Verification — https://huggingface.co/papers/2405.14092
[3] ExpeL: LLM Agents Are Experiential Learners — https://huggingface.co/papers/2308.10144
[4] Voyager: An Open-Ended Embodied Agent with Large Language Models — https://huggingface.co/papers/2305.16291
[5] Mem^p: Exploring Agent Procedural Memory — https://arxiv.org/html/2508.06433v2
[6] PATH-Bench: Path-Dependent Evaluation of Lifelong Agents — https://arxiv.org/html/2608.01149v1
[7] Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models — https://arxiv.org/html/2510.04618v1
[8] MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory — https://arxiv.org/html/2601.03192v1
[9] AgentCL: Toward Rigorous Evaluation of Continual Learning in Language Agents — https://huggingface.co/papers/2606.02461

---
Every [N] marker above is verifiable: fetch the citations JSON, match the marker id, and check the extraction (summary and key facts) plus placements against the source URL. Verification guide: https://lumisonde.com/for-agents/