How parallel text generation coordinates a passage

Introduction

A sentence can have every word in place and still contain a mismatch: “She are ready.” Generating several positions together makes this a concrete coordination problem. A model may consider “She” plausible at one position and “are” plausible at another, yet need an additional mechanism to choose a compatible pair.

In masked diffusion, the working passage is a sequence of visible tokens and special mask tokens marking unresolved positions. A token is a vocabulary unit, which can be a word or part of one. The model predicts possible replacements using the current passage; a separate generation procedure, the sampler, decides which values enter the next version. Repeating this operation makes more text available as context. In common absorbing-mask samplers, accepted tokens stay fixed even as predictions for the remaining positions change. [1, 2]

Gradual generation has two related jobs. Completion assigns values to the remaining positions. Agreement requires those values to respect relationships across the passage. Shared context can help dependent choices agree; commitment decides whether they get that context before becoming fixed; revision can let an earlier choice respond afterward. Following those operations—from learned recovery to the update rule and then the resulting text—shows what a claim of refinement promises and what its evaluation must check.

The model learns recovery; the sampler builds the passage

During masked-diffusion training, the original text supplies a known answer. A training procedure hides some tokens, gives the damaged sequence to the model, and scores the probabilities assigned to the original missing values. Different amounts of masking expose the model to different recovery problems: predicting from a few clues, or filling a small gap in an otherwise informative passage. MD4 and MDLM weight these recovery losses across masking levels, under their respective training formulations. [1, 2]

Consider a schematic training example, treating each word as one token: “The keys are on the table” becomes “The keys [MASK] on the [MASK].” The original “are” and “table” are the targets. The visible plural subject provides information about the missing verb.

Generation uses the learned predictions differently. There is no hidden original answer waiting to be uncovered. The sampler starts with masks in the region to be generated, asks for clean-token probabilities, and uses those probabilities to construct the next state. In MD4's basic reverse transition, a masked position either receives a sampled token or remains masked; an already visible token is copied. For infilling, supplied context stays unmasked. A particular transition can leave the state unchanged, so progress need not mean that every pass visibly fills a new blank. [1]

The loop keeps predictions and accepted values distinct. The current passage enters the predictor, whose probabilities support proposed token values. The sampler selects which proposals enter the next passage; that passage becomes input to the next evaluation. A proposal describes a possible choice, while a transferred value changes the context available afterward. The same trained predictor can therefore support different generation behavior when paired with different samplers.

Masks are one representation of uncertainty. In the continuous-state Diffusion-LM, generation starts from Gaussian noise vectors and repeatedly denoises them toward word representations. A learned mapping converts the resulting representations into words. Its clamping technique also maps predicted clean vectors to nearby word embeddings during intermediate sampling, but the next state is still a noisy vector state. Changing such a vector is different from editing an accepted word in a readable draft. The intermediate representation determines what “refinement” can change. [3]

A selected value becomes useful information for another position

For a masked model to use a later word when predicting an earlier gap, that word must be accessible to its predictor. LLaDA removes the causal attention mask that restricts a left-to-right Transformer. Its bidirectional attention lets visible text on either side contribute to a prediction. [4]

Timing matters within that access. During an ordinary model evaluation, positions exchange contextual representations while the network processes the current sequence. The sampler chooses token values afterward. A value inserted into the sequence then becomes input to the following evaluation. A still-masked verb can respond to the particular subject that was chosen, rather than only to the possibilities that existed before the choice. This input–prediction–update sequence is explicit in the masked sampling account used by PUNT. [5]

An idealized example separates the two situations. Suppose the only allowed completions of “[MASK] [MASK] ready” are “She is ready” and “They are ready,” each with probability one half. In this stipulated distribution, each blank considered separately has two equally likely values:

Independent draws give each row probability one quarter. Half the resulting pairs mismatch, even though both individual distributions are correct under the assumptions. If “They” is chosen first and supplied as context, the stipulated distribution for the verb becomes entirely “are.” What changes is the information available to the second decision. The calculation illustrates the distinction PUNT makes between error in learned conditional predictions and error from sampling dependent positions independently. [5]

A real sampler can try to discover which choices need that extra conditioning. PUNT proposes candidate tokens, orders their positions by confidence, and divides them into anchors and test candidates. It temporarily supplies anchor values, reevaluates the test-position distributions, and filters out candidates whose distributions change beyond a threshold. Those candidates can wait for a later iteration. The procedure asks whether proposed choices materially affect one another, rather than treating confidence at each position as sufficient evidence that they can be accepted together. [5]

These tests require extra model evaluations before tokens are committed. PUNT batches the tests, so the number of testing evaluations per iteration grows approximately with the logarithm of the number of masked positions. Its efficient construction relies on an independence-stability assumption, and its practical threshold tests use learned distributions to make approximate scheduling decisions. [5]

Other methods change how choices are coupled within an outer generation step. CoDiLA sends the diffusion model's local probability distributions to a small, separately trained autoregressive decoder. Within each short block, that decoder chooses tokens sequentially, conditioning later choices on earlier ones; different blocks retain an independence approximation. Thus a selected value need not always wait for the next full diffusion-model pass to influence another selection. Here the additional coordination happens inside a local decoder, with its own sequential work and restricted scope. [6]

The shared question is where a dependency gets resolved: through newly visible context on another pass, through tests that defer interacting proposals, or through explicitly coupled selection. Calling all of these operations a “step” hides the consequential difference.

A changing prediction is not necessarily a revised word

Once a prediction enters the passage, can a later pass replace it? The released LLaDA sampler makes the boundary inspectable. Its example caller tokenizes prompts, passes their token IDs and attention mask to generate, and decodes the returned response region. Inside generate, the working tensor x contains the prompt followed by masks. Each iteration records which positions are masked, evaluates the model, and forms proposed token values x0. In low_confidence mode, the sampler scores proposals by their model probabilities. [7]

The decisive update is the following exact excerpt from generate.py. Earlier proposal construction, confidence scoring and future-block exclusion are omitted; both supported confidence-ranking modes reach this same copy-and-transfer logic. [7]

x0 = torch.where(mask_index, x0, x)
confidence = torch.where(mask_index, x0_p, -np.inf)

transfer_index = torch.zeros_like(x0, dtype=torch.bool, device=x0.device)
for j in range(confidence.shape[0]):
    _, select_index = torch.topk(confidence[j], k=num_transfer_tokens[j, i])
    transfer_index[j, select_index] = True
x[transfer_index] = x0[transfer_index]

mask_index marks positions that were masks in the current x. The first line keeps new proposals there but copies the existing value everywhere else. The second gives visible positions negative-infinite selection confidence. The remaining lines select a scheduled number of proposals and write them into x, which is consumed by the next iteration and ultimately returned to the caller. Under ordinary non-mask prompts, accepted tokens and prompt tokens remain unchanged. The code's remasking='low_confidence' setting filters proposals for unresolved positions. [7]

A schematic state such as “They [MASK] ready” can therefore receive a different verb proposal on successive passes while “They” remains fixed. A proposal that was never transferred was never part of the passage. Withholding it gives a future prediction more context without revising an accepted word.

The editable region can also be smaller than the whole response. LLaDA's generate can progress through successive blocks, excluding future blocks from commitment until their turn. Its var_generate wrapper appends a block and passes completed output back as the next prompt. Blockwise generation concentrates iterative prediction inside successive regions, with earlier completed text supplying fixed context under the same visible-token preservation rule. [7]

Revision requires both permission and useful recovery behavior

ReMDM changes the transition rule itself. A decoded token can remain visible with probability 1 − σ_t or return to a mask with probability σ_t, where the remasking schedule controls that probability. Setting it to zero recovers the no-remasking case. A returned mask can subsequently be filled using the surrounding state, allowing a position that already had a value to receive another one. [8]

This spends computation on a different job from completing untouched blanks. ReMDM includes schedules that activate remasking later or loop at a fixed noise level, allocating extra work to reconsideration. Its OpenWebText experiments report sample-quality benefits under evaluated settings, using MAUVE to assess quality and diversity together. Its own analysis also shows that stronger remasking can improve generative perplexity while sacrificing diversity, reaching repetitive outputs at extreme settings. The ability to revisit a token creates a repair opportunity; selecting a useful amount of revisiting remains part of the method. [8]

Direct replacement is another route. The D3IM sampler reconstructs the next state from current predictions, retains a scheduled top-confidence set, and masks the rest. A visible position can consequently receive a different token without first becoming a mask. In the Revise, Don't Freeze preprint, however, applying this rewrite rule to the unadapted LLaDA-8B checkpoint harms performance on all four reported math and code benchmarks. [9]

That failure reconnects inference with training. In ordinary mask corruption, a visible training token is an original correct token. During generation, a visible token may be the model's own mistake. The preprint's SCOPE training constructs self-generated committed values and supervises both those positions and remaining masks against the original targets. Its paired training-and-sampler results improve in the evaluated settings, while the authors report that the benefit does not automatically transfer to other revisable samplers. The study uses LLaDA-8B and single-run results without error bars. The result ties the usefulness of revision to the model's behavior under its own generated context. [1, 2, 9]

Even a model allowed to refill a known error can reproduce it because of the remaining context. PURE investigates this with 160 incorrect GSM1K drafts from LLaDA-8B-Instruct, generated with a 256-token budget and 256 steps. [10] Researchers manually identify small answer-flipping token sets and allow ten refinement steps. Remasking only those locations is the least effective tested intervention; masking neighboring tokens and relaxing some preceding context improves correction. The authors describe the surrounding support for a wrong choice as contextual lock-in. [10]

PURE's practical procedure selects seed positions where confidence in an already committed token drops as surrounding context evolves. [10] It then remasks those positions together with local windows and a random subset of preceding context before refilling. This changes the evidence available for recovery as well as the token being reconsidered. Excessive disruption can also remove useful context. The mechanism reveals why richer context has two possible consequences: it can help dependent choices agree, or help an erroneous configuration reproduce itself. [5, 10]

Filling the canvas and satisfying its relationships are different outcomes

A stopping rule can establish that the requested positions have values. Evaluating agreement requires checking relationships among those values. Subject–verb agreement, valid code, consistent references across paragraphs and factual correctness ask different questions about the finished text.

ParallelBench makes this distinction experimentally concrete. Its main LLaDA-1.5 analysis compares parallel commitment policies across queue operations, text writing and puzzles. Copying is robust in its short-output setting, while random replacement and shuffling deteriorate strongly as parallelism increases: several individually plausible choices can violate a whole-output constraint. In sentence writing, tool-scored grammar degrades more sharply when the model must compose from sparse, unrelated required words than when it paraphrases informative input. Its main queue outputs have a 32-token limit; puzzles and text writing have a 64-token limit. [11]

The metric changes what a reader can conclude. ParallelBench reports that ROUGE-L and BERTScore, which compare outputs with reference text, are less sensitive to parallelism than required-word inclusion. A passage can remain similar to a reference while failing an exact content requirement. The metric determines which relationship in the finished text is being tested. [11]

PURE provides a small direct long-form probe with mixed results. On 50 constrained creative-writing prompts, it generates 1,024-token outputs and applies 100 post-hoc refinement steps. [10] Automatic checks assess hard requirements such as mandatory phrases and an exact final-paragraph sentence; GPT-4o-mini judges softer qualities and contradictions. The authors report a judged contradiction rate of 0.26 versus 0.30 for the comparison outputs, while the hard-constraint pass rate falls to 0.78 from 0.82. [10]

A revision can improve one relationship while disturbing another. Moving beyond local agreement therefore requires evaluations that check the document-level commitments the output must preserve, alongside the exact constraints that editing can break.

More coordination has a computational price

Once a quality requirement is chosen, the next question is how much computation it takes to meet it. For a fixed canvas whose tokens are each committed once, accepting larger batches leaves fewer intervening evaluations in which newly chosen values can inform the remaining positions. Whether that loss matters depends on the unresolved dependencies, the predictor and the task.

LLaDA's sampling-efficiency study varies 32, 64, 128 and 256 steps for a fixed 256-token output: respectively eight, four, two and one newly decoded tokens per pass. [12] The comparison uses LLaDA-8B-Base and LLaMA3-8B-Base in bfloat16 on one A100-80GB GPU at batch size one. [12]

The result is a task-dependent quality–speed trade-off. The study reports comparable math performance with higher throughput, comparable HumanEval performance at matched throughput, and weaker MBPP performance. [12]

Counting nominal steps also misses work inside them. PUNT adds testing evaluations; CoDiLA adds a local sequential decoder; revision can revisit positions that were already filled. PUNT's authors identify their principal advantage at low-to-mid model-evaluation budgets on longer-form tasks, with curves converging or crossing at larger budgets. An efficiency comparison must count the actual computation and measure output quality under the same generation conditions. The useful measure is the cost of producing text that meets the chosen quality requirement. [5, 6, 8]

Conclusion

Parallel text generation becomes a passage through repeated changes to what the model can use as context. Training supplies a recovery predictor. The sampler turns predictions into selected values, determines which choices become fixed, and may provide a route for reopening them. Agreement can improve when dependent decisions receive useful information before commitment or can respond effectively afterward.

The connection between completion and coherence is the information used for each commitment. Filling a position changes the context available elsewhere; reopening it changes that context again. The next question is which dependencies a procedure can recognize and coordinate at a worthwhile computational cost, especially across a whole document. Gradual generation makes those opportunities available through its prediction, commitment and revision rules.

Limitations

This account uses evidence available through 28 October 2026 and the specified paper revisions and repository commit. The code analysis establishes the inspected implementation's update path; it does not establish deployment behavior, and no model generation or benchmark was executed for this article.

The examples and methods are representative rather than an exhaustive taxonomy of diffusion language models. The strongest inspected outcome evidence concerns short constraints, sentence grammar, reasoning and code. The long-form probe is small, model-judged and incompletely specified for independent reproduction. Its mixed outcomes do not establish reliable document-wide consistency or factual correctness. Results from different models, hardware, output lengths and evaluation protocols do not support a single speed or quality ranking.