# How World Models Turn Experience into Practice

> How learned models let agents turn past observations into new attempts—and turn practice into reusable skills.

Published: 2026-10-19 · Lumisonde (https://lumisonde.com/reports/world-model-experience/)
Citations JSON (per-claim receipts): https://lumisonde.com/reports/world-model-experience/citations.json

## The brief

**Question:** Can AI learn to complete tasks by acting in a world it generates?

**Stake:** Collecting experience takes time and resources. If the same observations can support new attempts, the amount an agent learns need not be limited to actions it has already taken.

**Answer:** Yes: recorded observations can train a world model—a predictor of what actions will cause—and attempts inside it can improve the agent's rule for choosing actions.[1, 3] In Dreamer 4, this improves harder Minecraft skills beyond imitation without collecting new game interaction for learning.[1] The learned rule must still work outside the model: generated practice can exploit prediction errors, and continued outside experience can reveal where corrections are needed.[2, 9] This separates two training choices: which experience to acquire, and how much additional practice to draw from it.[1, 9]

## Introduction

An agent can learn useful skills by acting inside a learned model of the world.[1] The important change is how acquired experience becomes a resource for further learning: observations teach a model of consequences, generated attempts teach a policy, and outside interaction can test and correct both.[1, 9]

A recording of Minecraft contains a player's actions and what happened afterward.[1] An agent can imitate those actions.[1] But if it learns to predict how the game responds, the recording can also become the starting point for new attempts: choose an action, generate its consequences, choose again, and learn from how the attempt unfolds.[1] Dreamer 4 demonstrates this distinction. After learning from recorded gameplay, it improves several harder Minecraft skills through reinforcement learning inside its world model, without collecting new game interaction for learning.[1]

Some systems keep returning outside to correct the model; others perform this additional learning from a fixed dataset.[1, 9] To understand what this makes possible, we need to follow the experience rather than the scenery: how observations become a practice environment, how practice changes future decisions, and which learned skills survive execution beyond that environment. Minecraft supplies a particularly clear before-and-after experiment. Generated navigation worlds and physical robots then show how the same learning idea changes when tasks, feedback, and corrective experience come from different places.

## A world model turns observations into consequences

For this learning process, a world model is a predictor that responds to actions.[3] Given a representation of the current situation and an action, it generates a possible next situation.[3] Repeating that operation produces an imagined trajectory: a sequence of actions and consequences.[3] A reward predictor can supply task feedback, and a termination predictor can indicate that an episode has ended.[3] These components make the model usable as a practice environment rather than merely a producer of plausible pictures.

The practice environment does not have to produce pictures at all.[2] In the 2018 *World Models* study, researchers recorded 10,000 random-action episodes of VizDoom's Take Cover task, where survival requires avoiding fireballs.[2] They learned a compact numerical representation of images and a recurrent predictor of the next representation and whether the agent dies.[2] A controller was then optimized for survival entirely inside those predicted states, without rendering images during training, before being tested in the original game.[2]

That example exposes a useful separation between knowing consequences and knowing what to do. Random gameplay can help teach how a scene changes even though it does not demonstrate a good survival strategy.[2] The controller's subsequent optimization searches for useful behavior within the learned account of the game.[2] In this arrangement, the outside experience supplies the model's grounding; generated experience supplies additional opportunities to discover an effective policy.[2]

Other interfaces make the generated world visible. DIAMOND uses a diffusion model to generate the next game image from recent observations and actions.[5] World-Gymnast rolls a robot policy through an action-conditioned video model.[7] The learner can therefore practice on generated images, not only on hidden numerical states. These are different implementations of the same consequential relationship: the learner's action changes what it encounters next.

## Practice changes a policy, not just a prediction

A world model can predict a successful outcome without supplying a decision rule that reaches it. To turn generated experience into a skill, training must change the policy—the rule that selects actions from the agent's observations or internal state. DreamerV3 makes this bridge explicit through three components: a world model predicts consequences, an actor selects actions, and a critic estimates future cumulative task reward.[3]

Training begins from a state encoded from stored experience.[3] The actor samples an action; the model predicts the next state and reward; the actor acts again.[3] The critic learns from the resulting sequence.[3] The actor is then updated to favor sampled actions whose estimated return was better than its baseline expectation, and to discourage those that were worse, while retaining an incentive to explore.[3] The next imagined attempt is consequently made by a changed learner. During actual environment interaction, DreamerV3 selects actions with this trained actor rather than searching through future action sequences at every step.[3]

Delayed rewards make the critic important. A useful action may produce no immediate task reward but lead to a situation from which success becomes more likely. DreamerV3 combines predicted rewards along a short rollout with estimates of what comes afterward; the cited version uses a 16-step prediction horizon.[3] Short generated attempts can therefore contribute to learning behavior whose eventual payoff lies farther ahead.[3]

Dreamer 4 uses a related division of labor with a different policy-update rule.[1] In its principal imagination-training setup, the learned predictor stays fixed while the action rule and the estimator of future task reward improve.[1] Generated trajectories receive rewards from an already learned reward predictor.[1] The policy update balances better-than-expected and worse-than-expected samples and constrains movement away from a frozen copy of the imitation policy.[1]

This persistent learning differs from using a model only to plan the next move. PlaNet, for example, learns dynamics and rewards but has no separate policy or value network.[4] It searches candidate action sequences, executes the current selected action, and replans after receiving another observation.[4] Both approaches use learned predictions to act, but only the former examples establish that generated practice trained a reusable actor.[3, 4] Asking what changes during practice distinguishes a learned skill from a good decision found by search.

## A fixed Minecraft dataset can teach more than imitation

Dreamer 4's Minecraft experiment is especially informative because its starting policy already benefits from both demonstrations and world-model learning.[1] The researchers use 2,541 hours of contractor gameplay containing video, actions, and event annotations.[1] The model first learns action-conditioned prediction, then a task-conditioned imitation policy and reward model.[1] Twenty tasks and their sparse rewards are derived from recorded game events.[1] Only afterward does reinforcement learning improve the policy on model-generated trajectories.[1]

The useful comparison is therefore not an untrained agent against a fully trained one. The paper's “WM+BC” baseline already has the world-model representation and behavioral cloning, meaning training to imitate recorded actions.[1] The full agent adds imagination reinforcement learning.[1] It tests whether generated practice adds something beyond the predictive representation and imitation policy that are already present.

The resulting agents are evaluated in Minecraft itself, not in the learned simulator.[1] Episodes start in randomly generated worlds with empty inventories and last up to 60 minutes.[1] Both agents receive a supplied sequence of task prompts, progressing through gathering materials and crafting tools toward diamond mining.[1] Appendix D reports these milestone success rates over 1,000 evaluation episodes per agent:[1]

| Minecraft milestone | World model + imitation | After imagination RL |
| --- | ---: | ---: |
| Obtain a log[1] | 99.6% | 99.1% |
| Obtain a stone pickaxe[1] | 89.4% | 90.1% |
| Obtain an iron pickaxe[1] | 16.9% | 29.0% |
| Obtain a diamond[1] | 0.0% | 0.7% |

Dreamer 4's milestone table supplies no uncertainty estimates for the cited before-and-after differences.[1] The iron-pickaxe result makes the additional practice tangible: success rises from 16.9% to 29.0% while the dataset stays fixed.[1] Recorded experience has already taught the agent a representation and an imitation policy; generated attempts then provide another way to improve its decisions.[1] The larger gain at this harder milestone is especially useful for understanding what the extra training contributes.

The finding changes how to think about a fixed dataset. Alongside its recorded action sequences, it can support attempts made by the current policy and feedback on their generated outcomes.[1] The agent can revisit situations with a different way of acting and learn from the consequences predicted by its model.[1] A collection of past interactions becomes a resource for further practice.

DIAMOND extends this picture to image-based practice across 26 Atari games.[5] Its Atari 100k experiment trains a separate agent for each game, with up to 100,000 original-environment actions per game and five training seeds.[5] An actor-critic learns inside the diffusion world model, while additional original-game experience periodically updates the model.[5] The paper reports performance above its human reference on 11 games.[5] Here the training process alternates between acquiring more game experience and learning from generated attempts based on what has been acquired so far.[5]

## A generator can supply a curriculum

Once a learner can improve through practice, a second question appears: which situations should it practice in? A learned generator can supply environments to another agent, while a task setter chooses goals and a reward model scores attempts.[6]

SIMA 2 explores this arrangement with worlds generated by Genie 3.[6] Genie provides interactive environments controlled through keyboard and mouse actions.[6] SIMA 2 is the learner, initialized from a Gemini Flash-Lite model and trained using a mixture that includes substantial human gameplay demonstrations and synthetic interaction data.[6] A Gemini task setter proposes instructions suited to the current situation; a separate Gemini reward model scores videos of the agent's attempts.[6] Training on that scored experience changes the agent.[6]

For the further-training experiment, urban environments and tasks supply practice, while natural environments and tasks are held out.[6] The tasks are primarily navigation.[6] Evaluation uses Gemini's 0–100 rubric scores for completion and directedness, calibrated against a small dataset of human preferences.[6] On this measure, the authors report improvements on nearly all training tasks and a majority of held-out test tasks, including navigating toward a red mushroom in a natural scene.[6] The reported quantity is a model-assigned rubric score rather than a directly measured task-completion rate.[6]

Curriculum design could combine generated settings with the SIMA framework's described ability to steer practice toward weaker skills.[6] SIMA's task setter can direct attention toward weaker skills, while its generator supplies varied surroundings.[6] As competence grows, a promising next question is how to choose situations that keep teaching the agent something useful. World generation opens a way to organize practice around the learner's changing abilities.

## What carries back to a robot—or fails to

Physical robots provide a consequential destination for generated practice: a trained policy must select actions that work when the predicted images are replaced by camera observations and the imagined arm by a real one. World-Gymnast starts from an OpenVLA-derived policy already fine-tuned on Bridge V2 demonstrations.[7] It then samples robot actions, generates their visual consequences in WorldGym, and uses GPT-4o's binary task-completion judgments for reinforcement-learning updates.[7]

The researchers compare this against the starting supervised policy and an iterative imitation approach trained on successful synthetic rollouts with an equal rollout-step budget.[7] They execute the trained policies on a physical robot using AutoEval, an automated system with a scene-trained success classifier and reset policies.[7, 8] The protocol uses five groups of ten trials per policy-task; the reported errors are standard errors between groups.[7]

For putting an eggplant into a blue sink, physical success rises from 4 ± 4% for the starting policy to 72 ± 10% after World-Gymnast training; iterative synthetic-data imitation reaches 10 ± 3%.[7] In this case, the additional practice happens through generated visual consequences, and the resulting policy selects actions for a real arm.[7] The experiment connects the two ends of the learning process: simulated attempts during training and successful physical execution afterward.[7]

Why can additional practice fail? The earlier Take Cover experiment supplies an unusually direct answer. The *World Models* controller found movements that suppressed or extinguished fireballs inside the learned environment.[2] At a low sampling temperature, the generator often failed to produce fireballs at all.[2] The resulting controller averaged 2,086 survival steps in the imagined world but only 193 in the original game, below the random policy's 210.[2] Making the training world more stochastic produced a much stronger original-game controller, although further increases in randomness did not keep improving its score.[2]

The lesson concerns what optimization learns. Repeated practice strengthens behavior that succeeds under the available consequences and rewards. If the model removes a danger, the policy can learn to rely on that removal. If it preserves useful action consequences, practice can teach a portable strategy. A high imagined survival score could therefore reflect exploitation of the model's missing fireballs rather than a strategy that worked in the game.[2]

Practice allocation also shapes which abilities develop. In World-Gymnast's main experiment, drawer-closing success stays at 62 ± 6%; further imagination training targeted to a particular test scene raises it to 100 ± 0%, while performance on other tasks generally degrades.[7] This creates a concrete curriculum question: how should practice be distributed so that improving one skill also preserves the others?

## Scaling practice changes the role of outside experience

That practice-allocation question remains open. A separate question is when the learner needs new outside observations. Generated practice and outside interaction need not be competing sources of learning. DayDreamer runs them as a continuing loop on physical robots.[9] Real actions and their observed outcomes enter a replay buffer.[9] A learner fits a world model to that experience and trains an actor and critic on predicted trajectories, while another thread operates the robot and supplies more data.[9] Actor and critic gradients do not update the world model; the authors separate them to avoid pushing its predictions toward overly optimistic outcomes.[9]

Here, outside experience does more than initialize a practice world. It can reveal where the current account has stopped working. DayDreamer's XArm performance collapsed under a sharp lighting change and recovered with further physical training.[9] Strong light rays across the scene made the robot's image observations differ greatly from its original training observations.[9] The recovery illustrates how continued learning from physical experience can adapt behavior to changed visual conditions.[9]

World-Gymnast describes a related iterative extension: collect fresh physical trajectories, update the world model, and use it for further policy training.[7] Its reported correction phase collects around 100 physical trajectories per task.[7] Dreamer 4 instead demonstrates useful additional learning while keeping its outside dataset fixed.[1] These regimes answer different questions: how far can existing experience be exploited, and when does better learning require new observations?

This suggests two linked budgets for an agent-learning system. One is the budget for acquiring experience from the target environment. The other is the computation spent exploring and learning from a predictive account of that experience. A learned world model makes those budgets partly separable: the agent can attempt more than it physically executes, and retain the resulting lessons in a policy.[1] New observations can then reveal where that predictive account needs revision.

This division suggests a way to scale learning through more purposeful practice. With a model learned from the experience already available, a learner could spend additional computation on difficult starting states, alternative actions, or underdeveloped skills. A curriculum could adjust those choices as the policy improves, while outside interaction supplies observations that sharpen the next round of practice. How to allocate these two budgets—acquiring experience and learning more from it—is a research question that connects training scale, curriculum design, and capability development.

## Conclusion

World models are changing how AI can acquire and use experience. Recorded interactions can teach a model of consequences; acting within that model can turn those consequences into skills.[1] Dreamer 4 shows how a fixed gameplay dataset can support additional practice.[1] SIMA 2 explores generated settings and goals as a curriculum.[6] World-Gymnast and DayDreamer connect model-based practice with physical action and fresh observations.[7, 9]

Together, these systems reveal a productive division of labor. Outside experience establishes and revises the agent's understanding of the world. Generated experience gives it room to explore the action consequences of that understanding and develop strategies it can use again. How best to organize that practice remains a research direction.

## Limitations

The results discussed are author-reported experiments. Their games, policies, data budgets, and evaluators differ, so they do not form a matched ranking. Gains vary by milestone, and diamond success remains rare under the supplied task prompts.[1] SIMA's generated-world learning is navigation-heavy and evaluated through a Gemini rubric.[6]

World-Gymnast's physical trials use small evaluation groups and an automated judge; AutoEval's original study documents occasional scoring and reset failures.[7, 8] World-Gymnast's main tables include a fourth task, eggplant-to-yellow-basket, while Appendix C.6 describes only three physical tasks; the individual results cited above do not resolve that scope discrepancy.[7] The error bars describe evaluation groups rather than independently trained seeds.[7] The studies examine particular training runs; the scaling and curriculum possibilities discussed here remain research directions.

## References

[1] Training Agents Inside of Scalable World Models, arXiv v1 — https://arxiv.org/abs/2509.24527v1
[2] World Models, captured paper text — https://huggingface.co/papers/1803.10122
[3] Mastering Diverse Domains through World Models, arXiv v2 — https://arxiv.org/abs/2301.04104v2
[4] Learning Latent Dynamics for Planning from Pixels, arXiv v5 — https://arxiv.org/abs/1811.04551v5
[5] Diffusion for World Modeling: Visual Details Matter in Atari, arXiv v2 — https://arxiv.org/abs/2405.12399v2
[6] SIMA 2: A Generalist Embodied Agent for Virtual Worlds, arXiv v1 — https://arxiv.org/abs/2512.04797v1
[7] World-Gymnast: Training Robots with Reinforcement Learning in a World Model, arXiv v1 — https://arxiv.org/abs/2602.02454v1
[8] AutoEval: Autonomous Evaluation of Generalist Robot Manipulation Policies in the Real World, arXiv v2 — https://arxiv.org/abs/2503.24278v2
[9] DayDreamer: World Models for Physical Robot Learning, arXiv v1 — https://arxiv.org/abs/2206.14176v1

---
Every [N] marker above is verifiable: fetch the citations JSON, match the marker id, and check the extraction (summary and key facts) plus placements against the source URL. Verification guide: https://lumisonde.com/for-agents/