How Robots Turn Language into Action

Introduction

Imagine asking a robot, “Put the cup in the sink.” The sentence leaves out which cup, where to grasp it, how to move around the counter and what to do if the cup slips. A person fills in those details almost without noticing. A robot needs to connect them: identify the cup, choose a grasp, move the arm and check whether the gripper has hold of it.

Robots connect these decisions in several ways. A language model can choose among existing skills, generate a program that calls perception and control functions, or participate in a learned policy that predicts numerical actions from instructions and observations. A policy is the rule or model that chooses actions from the information available to it. These are alternative interfaces that can also be combined.[1, 2, 3]

The shared story is a loop. The request and observations constrain what the robot should do; an executable behavior produces commands in the robot's action interface; movement changes the world; new evidence can change the next decision. Training and engineering supply the abilities used inside that loop. Following these connections makes it possible to read a robot demonstration more precisely: what resolved the request, what actually commanded the body, and what evidence could make it change course?

Connecting the request to the scene

The cup example contains two different questions. Which behavior would advance the request? Which behavior can this robot perform from its present state? A system called SayCan makes that distinction explicit. Each available skill has a textual description, an executable policy and an estimate of whether the skill will succeed now. The language model scores how useful the description is for the instruction, taking earlier selected descriptions into account. The system multiplies that score by the state-dependent success estimate and selects the highest-scoring skill.[1]

For an illustrative request to fetch a drink, picking up a drink can be relevant while remaining difficult from the robot's current position. Moving toward its location can make a later pick more feasible. In SayCan's actual loop, the chosen policy executes, the robot state is updated, the chosen description enters the planning history, and the system scores the next options. Its “affordance” estimate—the estimate that an interaction is possible from this state—connects verbal relevance to available behavior. The estimate helps select which executable skill to try next.[1]

Another approach gives the language model a more flexible interface than a list of named skills. In Code as Policies, the model receives examples and descriptions of available programming functions. Its generated program can obtain an object's position, calculate an offset and pass the resulting target to a pick-and-place function. Conditions and loops can make later calls depend on perception. For instance, “a little to the right” becomes a spatial calculation under the supplied coordinate conventions. The generated program composes these available perception and control functions.[2]

A vision-language-action model, or VLA, can instead condition numerical action prediction directly on an image and a task instruction. RT-2 reformulates robot control in this way: a camera observation and text prompt lead to action tokens. An RT-2 variant also generates a short natural-language plan before its action tokens. Explicit reasoning and learned action prediction can therefore coexist inside the same approach.[3]

If two cups fit the request, the robot may need the person to choose between them. KnowNo adds an explicit act-or-ask decision: it scores candidate next steps, uses held-out examples to calibrate a prediction set, and acts when that set contains one option; otherwise it asks a supervisor to resolve the choice. Its statistical task-completion guarantee assumes the relevant environment is correctly represented in text and the proposed actions execute successfully. Clarifying intent and carrying out the answer are separate responsibilities.[4]

Giving an action physical meaning

Once the robot has a situated objective, something must specify movement. The robot's end effector is the tool at the end of its arm, such as a gripper. In RT-2's 2023 formulation, an action describes six dimensions of end-effector displacement—translation and rotation—plus gripper extension and a termination command. Continuous dimensions are divided into 256 bins and represented with reserved vocabulary tokens.[3] During robot action prediction, decoding is restricted to valid action tokens.[3]

The tokens compactly represent the requested movement and gripper setting. The termination command signals that the policy considers the episode finished. The robot's control system executes the movement, and subsequent observations show what happened.

Other policies generate continuous action sequences. In the π0 v4 paper, the model receives images, language and robot joint angles and predicts a chunk of 50 future actions. During training, demonstrated action chunks are mixed with noise, and the model learns how to adjust these noisy candidates toward the demonstrated actions. At inference, it starts with random noise and repeatedly uses that learned guidance to refine a candidate action sequence. The guidance is a vector field; the repeated refinement is numerical integration. The reported ten integration steps refine the predicted sequence before execution.[5]

Following a chunk into a real robot interface

A separate implementation example comes from the released openpi DROID client. It shows how a predicted chunk reaches a robot interface configured for seven joint-velocity commands and one gripper-position command. At a prediction boundary, it sends the instruction, an external-camera image, a wrist-camera image, joint positions and gripper position to a policy server. In the inspected client, the returned action array must have shape (10, 8): ten time steps, each containing eight command values.[6]

The client then consumes one row at a time. This exact excerpt shows the selected row becoming the vector handed to the environment:

# Select current action to execute from chunk
action = pred_action_chunk[actions_from_chunk_completed]
actions_from_chunk_completed += 1

# Binarize gripper action
if action[-1].item() > 0.5:
    # action[-1] = 1.0
    action = np.concatenate([action[:-1], np.ones((1,))])
else:
    # action[-1] = 0.0
    action = np.concatenate([action[:-1], np.zeros((1,))])

# clip all dimensions of action to [-1, 1]
action = np.clip(action, -1, 1)

env.step(action)

Source: openpi DROID client, action-execution loop.[6]

The last component is thresholded into a binary gripper command; all components are clipped before env.step consumes them. The environment interprets the seven arm components as joint-velocity requests. A joint is a movable connection between arm segments; a velocity request specifies its movement rate over the execution interval. Coordinating those joint motions can change the gripper's position. Clipping keeps all eight requests within the permitted numerical range.

The matching checkpoint and adapter matter as much as the vector's length. openpi's action-space documentation describes joint-angle conventions for several robots but explicitly specifies joint velocities for DROID, with its own control frequency. Normalization statistics are stored with checkpoints and must match the action definitions. Treating a velocity vector as a position vector would change the operation being requested, even if both arrays had the expected shape.[7]

Where planning and controllers can enter

A geometric goal can also pass through a separate motion-planning stack. MoveIt illustrates this route: a request specifies a joint configuration or end-effector pose; a planner searches for a path under the configured constraints and planning-scene collision model; postprocessing adds timing, velocities and accelerations. Its documentation notes that time-optimal trajectory generation can alter waypoints within a tolerance, potentially requiring further collision checks. A checked path and its timed execution are distinct artifacts.[8, 9]

Timed joint waypoints then need a controller to turn them into commands over time. For example, the ros2_control joint trajectory controller interpolates between them. When configured to command positions, it forwards desired positions. When configured to command velocities or effort, it can use the difference between desired and measured motion to correct tracking.[10]

Across these routes, the decisive handoff is the same kind of agreement: the producer's output must mean what the next consumer expects. An object location, a gripper displacement, a joint velocity and a timed joint trajectory are different objects with different downstream work still to do.

Where executable ability comes from

The action interface explains what a model can command. Training explains why it chooses useful commands. OpenVLA offers a direct example. It starts from a pretrained vision-language model and trains on robot trajectories pairing observations and instructions with actions. Each continuous action dimension is discretized into 256 bins, and training penalizes incorrect predicted action tokens.[11] Repeated examples connect what the robot sees and is asked to do with the movements demonstrated in that situation.[11]

The inherited visual and language representations provide reusable knowledge, while robot experience connects that knowledge to action. This division is visible in OpenVLA's adaptation experiments: further training on target-task demonstrations helps it operate in new Franka setups. Its “scratch” comparison retains the pretrained vision-language backbone while omitting broad robot pretraining, so that comparison tests the contribution of robot experience. The same study finds a task-specific Diffusion Policy competitive or better on narrow single-instruction tasks. Broad prior experience and specialized precision are useful under different conditions.[11]

Executable behavior can also be engineered or improved through rewards. SayCan's deployed repertoire mixes learned picking, scripted/motion-planned placing and classical navigation. The language-level selector can use them through a common interface even though their abilities came from different sources.[1]

In reward-based learning, the robot's consequences help change the policy itself. SERL, a system for learning manipulation skills, stores experience, trains a critic to estimate the reward associated with actions, and updates an actor to favor actions the critic values. Demonstrations can seed that process; new trials expose what happens when the learned policy acts. State tests or image-based success classifiers can supply the reward signal used for learning.[12]

Learning from how a scene changes

Video offers another kind of prior experience: how objects and bodies move over time. DreamZero, introduced in February 2026, starts from a pretrained video diffusion model and learns to generate future visual representations and robot actions jointly, conditioned on language, observations and robot state. One model denoises the two together. The authors interpret this joint visual prediction as implicit planning.[13]

Learning from future prediction, consulting future-related internal features and producing future images are distinct operations. Fast-WAM uses future-video prediction during training to shape shared representations, while keeping action prediction from reading the predicted future; at deployment it removes future generation and predicts actions from the current context. Its controlled comparisons on LIBERO, RoboTwin and a real towel-folding task support a substantial contribution from predictive training.[14]

Faster-WAM retains future-related context at deployment in a different form: the video component processes a noisy representation of the future once, and action generation reuses the resulting internal features. Under LIBERO-Plus visual shifts, its controlled comparison finds better performance with this future-related context than with current context alone.[15] Useful physical knowledge can therefore enter the action policy through more than one operation.

Letting consequences change the next decision

Suppose the robot starts reaching and the cup shifts. A fresh camera image can matter only if it reaches a decision the robot can still revise. Predicting a long sequence therefore raises a practical question: how much of that prediction will execute before the next update?

Diffusion Policy separates three quantities: the history of observations used as input, the number of future actions predicted, and the number executed before replanning. It can predict farther ahead than it commits to moving. After executing a prefix, it uses updated observations to predict again. This receding-horizon loop revises movement from visual input.[16]

The DROID client makes the distinction inspectable. Its default is to execute eight rows from the ten-row prediction before requesting another chunk, pacing execution toward 15 Hz. Eight nominal periods are about 0.53 seconds, calculated as 8 / 15; inference and transport can add time. Although the client reads observations every iteration, those observations enter a new action prediction only at the chunk boundary. Capturing an image and using it to revise movement are different events.[6]

A robot can also compute a replacement while continuing to move. Real-Time Chunking, or RTC, generates the next chunk asynchronously. It conditions that generation on the prefix of old actions that must still execute during the computation delay, then guides the overlapping remainder to preserve continuity while allowing later actions to change. This matters when successive predictions choose different ways around an obstacle: simply switching between them can produce a sharp, incompatible movement. RTC's committed prefix gives the new prediction a starting obligation.[17]

There are several clocks in this process. A low-level controller can track targets faster than the visual policy produces new ones. In Diffusion Policy's real Push-T setup, 10 Hz commands are interpolated for a 125 Hz robot interface.[16] The visual-policy update and the controller's target-tracking update run on their own clocks. Reaction depends on observation age, computation, communication and how commands are consumed.

Revising movement is different from recognizing success

After the gripper closes, the next consequential question may be whether the cup was picked up at all. Inner Monologue supplies such information to a language-level planner: a skill-success report and scene information are converted into text and added to its prompt. The planner can then retry a failed skill or choose a different next step. The information providers vary by experiment. The kitchen system uses a learned visual success detector and human-provided object feedback; its open-ended question-answering examples use human answers.[18]

Those paths explain both recovery and its failures. In the paper's real tabletop experiments, removing success feedback prevents retries after failed pick/place actions under injected noise. The paper also documents false success reports that hide failures, false failure reports that provoke unnecessary retries, and planners ignoring feedback. Recovery requires relevant evidence, a useful interpretation of that evidence and an available corrective behavior.[18]

Predicted futures have a different role from evidence of consequences. DreamZero updates its stored visual context with actual observations after execution and discards the predicted video latent. Its failure examples include picking up bread before opening an oven and then becoming stuck with the gripper occupied. The executed movement can agree with the predicted visual sequence while the sequence itself fails to accomplish the request.[13]

These adjustments also differ from learning. A fixed policy can map a changed image to a different action using the same weights. In SERL, a learner explicitly updates parameters from experience while an actor chooses actions.[12] New inputs change the current decision; learning changes the machinery that will make later decisions.

Why familiar words become difficult in a new situation

The same request can demand different abilities after a change of object, camera or body. A different cup may require a different grasp; a moved camera changes the relationship between pixels and motion; a longer arm changes reach and usable strategies. In π0.7's cross-embodiment experiments, the larger, heavier UR5e arms have different geometry, placement and grippers from the source robots, so transferring a task requires more than copying a trajectory.[19]

The April 2026 π0.7 paper also makes the contribution of language especially tangible.[19] The model uses varied training context, including detailed instructions, episode metadata and optional subgoal images, to steer action. On some short tasks, the authors report useful behavior without deliberately collecting robot data for that task. For unfamiliar multi-stage tasks such as loading an air fryer, however, one coarse request is insufficient. A person coaches the robot through intermediate instructions; those coaching episodes can subsequently train a high-level language policy for autonomous execution without new low-level teleoperation demonstrations.[19]

That is a meaningful change in how a task can be taught. A human supplies missing high-level choices through words because the robot already has reusable lower-level abilities. The later autonomous policy adds learned sequencing. The paper also acknowledges that large mixed datasets make truly unseen tasks hard to certify: related behavior may have appeared incidentally or under another label. Its result is best understood through what supervision was supplied and which decisions it enabled.[19]

Controlled tests can expose which connection remains fragile. In the CVPR 2026 LIBERO-Plus simulation study, OpenVLA-OFT scores 97.1% task success on unperturbed inputs, 56.4% under camera perturbations and 31.9% under initial-robot-state perturbations.[20] These are separate conditions in the same study, showing that strong familiar-setting performance can coexist with sensitivity to how the scene is viewed or where motion begins.[20]

Language needs its own intervention. In that study's object suite, removing the instruction leaves OpenVLA-OFT's performance largely unchanged; in the layout suite, changing the requested target and corresponding task goal produces major failures. Stability under paraphrases alone would miss that distinction: a robot can appear insensitive to wording because it follows a familiar scene-to-action pattern. Changing the goal tests whether the words actually change what it does.[20]

The useful question about generalization is therefore specific: what changed, what prior experience remained useful, what adaptation or human help was added, and was the outcome partial progress or completed work? Those conditions locate the connection that transferred—and the one still needing support.

Conclusion

A robot turns language into action by making an outcome actionable in a particular situation. Some systems expose skill choices or generated programs; others learn numerical action prediction, sometimes alongside language subtasks or visual futures. Each route still depends on executable abilities, a matching body interface and a way for consequences to influence later decisions.

The distinction between recognizing a goal and accomplishing it becomes clearest at the handoffs. Identifying the cup leaves contact and movement to solve. Producing a valid command leaves execution to solve. Executing a movement leaves its contribution to the task to assess. Progress comes from strengthening these connections together—giving the robot ways to finish the task and to recover when the world differs from its prediction.

Limitations

This account centers on language-conditioned manipulation, with selected navigation, planning and controller examples. Each method is presented within its own robot, task and evaluation protocol. The DROID implementation trace reaches its robot-environment interface; actuator-level units, scaling and command conversion require separate driver inspection. The π0 mechanism and prediction horizon described here follow the v4 paper, while the DROID example follows its released client configuration.

The research establishes specific ways to clarify, predict, execute and recover. Extending these abilities across people, objects and disturbances calls for whole-task evaluation of the underlying dependencies: execution assumptions in calibrated planning, scene coverage in collision checks, and the accuracy of learned success detectors and visual futures.