How speculative decoding checks ahead—and when it saves time

A language model usually generates text one token at a time. A token may be a word, part of a word, or punctuation. To choose the next one, the model uses the history already written; choosing that token establishes the history for the following choice. Even fast hardware must wait for those choices to arrive in order.

Speculative decoding changes what is available before the expensive model runs. A cheaper drafter proposes a continuation. The larger target then evaluates the candidate histories together, and an acceptance procedure commits a valid prefix plus a target-supplied correction or extra token. Several output tokens can therefore share one target round. The target still determines the intended generation behavior; the draft supplies opportunities to do its work ahead of time. [1, 2]

Three connected questions decide whether this accelerates delivery: which candidate histories can be checked together, which checked tokens can survive, and whether the surviving output repays the work. DFlash makes block proposals cheap using target context features. DSpark adds inexpensive dependencies between proposed tokens and allocates verification effort. DLoop spends additional drafting stages to postpone a target check. Following these decisions shows how proposal cost, prefix survival, and target-round cost combine to determine acceleration. [3, 4, 5]

A known candidate history can be evaluated before it is accepted

Let H be the accepted text, and let a drafter propose three token IDs, a b c. The target can evaluate all four of these conditional distributions:

During ordinary generation, H a is unavailable until the model chooses a. During verification, the drafter has already supplied a, so the target can compute its prediction after H a without waiting to choose a itself. The original speculative-decoding and speculative-sampling algorithms explicitly evaluate these extended prefixes in parallel. [1, 2]

A causal Transformer makes this possible without letting later tokens influence earlier predictions. Within each layer, the supplied positions can be processed together, while attention restricts each position to its allowed history. The output after input a predicts what follows H a; it checks b, not a. There are still sequential layers inside the model, and each prediction still depends on its preceding candidate tokens. What disappears is the need to complete a separate target invocation and token choice before starting each later prediction. [2, 6]

Scoring is only the first half of verification. For a schematic greedy example with consistent decisions and compatible numerical evaluation, suppose the target's highest-probability choices match a and b, then prefer d over c. The system commits a b d. The distribution already computed after H a b supplies d in the same target pass. If all three candidates match, the final distribution can supply a bonus token, subject to stopping conditions. [1, 2]

Replacing a candidate also determines which later computations lose their use. A score computed after H a b c describes that history, not the corrected history H a b d. The suffix after the first rejection must be discarded for this continuation. Parallel evaluation has made several conditional answers available; commitment determines which questions actually became relevant.

One concrete implementation makes that boundary inspectable. In Transformers v4.46.0, the ordinary greedy assisted-decoding branch compares the proposed new tokens with all but the last target prediction:

candidate_new_tokens = candidate_input_ids[:, cur_len:]
n_matches = ((~(candidate_new_tokens == selected_tokens[:, :-1])).cumsum(dim=-1) < 1).sum()

# Ensure we don't generate beyond max_len or an EOS token
if is_done_candidate and n_matches == candidate_length:
    n_matches -= 1
valid_tokens = selected_tokens[:, : n_matches + 1]

Here selected_tokens contains the target's next-token choices. The cumulative mismatch count keeps matches only before the first disagreement; n_matches + 1 includes the target's correction or bonus. Earlier in this function, one target forward call produces candidate_length + 1 relevant rows of logits—token scores before probability normalization. The excerpt omits the separate stochastic rejection-sampling branch. [6]

In this inspected implementation's cache bookkeeping, the code retains processed context through the accepted prefix and removes speculative state beyond it. Its final correction or bonus token is committed text but has not yet been processed into the target's attention cache. Thus a check of three candidates can process the preceding round's final correction or bonus token alongside those candidates to obtain four next-token predictions. Emission advances one token beyond the processed cache: the next round carries that last committed token into the target alongside its new proposals. [6]

Random sampling preserves the target's probabilities

Greedy matching is easy to picture because the target has one preferred token. Randomized generation instead asks for samples from a probability distribution. In the classical exact speculative-sampling rule, let p(x) be the target distribution and q(x) the distribution that actually produced the proposal, both at the same history. These distributions include the chosen temperature or other sampling transformations. [1, 2]

Draw x from q, then accept it with probability:

min(1, p(x) / q(x))

At the first rejection, sample a replacement from the normalized positive part of p − q, and discard the remaining draft. If every proposal passes, sample the bonus from the final target distribution instead. [1, 2]

The reason for the correction is a simple accounting of probability mass. Accepted proposals contribute q(x) × min(1, p(x)/q(x)) = min(p(x), q(x)). Rejection followed by residual sampling contributes the missing max(0, p(x) − q(x)). Together they reconstruct p(x). To illustrate the proof, suppose the drafter proposes a token with probability 0.4 and the target wants probability 0.2. Accepting half those proposals supplies that token's required mass. The residual supplies tokens the drafter underrepresents. [1, 2]

This particular verifier needs the actual proposal probabilities. A sequential head, parallel generator, or refinement procedure must supply the normalized probabilities of the sampling process it really used for the acceptance ratio and residual. There are other valid couplings: the inspected Transformers function also has a path that draws target tokens for candidate histories and continues only while they equal the proposals. At the first mismatch it emits the target draw. This gives a different acceptance behavior from the ratio-and-residual method: whether a match occurs changes how far the round continues, while the emitted current token is the target sample either way. [1, 6]

These guarantees concern the specified target distribution, within implementation numerics. They do not require two random runs to produce the same sentence, even with the same nominal seed: speculative execution consumes random numbers differently. Different computation graphs can also change floating-point results, and a single changed greedy token can send later text along another path. Equal sampling distributions, identical deterministic decisions, and similar task scores are distinct claims. [2]

With an appropriate verifier, inaccurate proposals cost drafting and verification work while the target retains its intended output distribution. That wasted work makes the design of the drafter consequential.

Cheap proposals need a coherent prefix

The straightforward drafter is a smaller autoregressive model: it proposes several tokens sequentially before the target checks them. That saves target rounds but adds a chain of draft-model passes. A block-parallel drafter moves the expensive representation work for several proposal positions into one pass. The next problem is making those simultaneously prepared positions agree on a continuation. [1, 3, 4]

DFlash reuses the target's context work

DFlash conditions its lightweight drafter on context work the target has already performed. The target first processes the prompt and supplies an initial token. Hidden representations from selected target layers are concatenated and projected into a compact context feature. Each draft layer turns those features into attention keys and values, caches them, and lets the draft positions attend to them. The keys and values are the stored representations attention reads when drawing information from the context. [3]

The drafter receives a clean anchor token, normally supplied by the preceding target round, followed by masked positions. It predicts the masked positions together in one block-diffusion pass, with bidirectional attention inside that block. Its training aligns these predictions with a frozen target and prevents information from leaking between training blocks. This combines two savings: reuse the target's context representations, and avoid rerunning the drafter's backbone—its main representation-producing network—for each proposed token. [3]

The interface matters operationally. Integrating DFlash requires access to target internals and draft-side caches. Its training description counts the anchor inside block_size, yielding block_size − 1 new predictions. DSpark changes the prediction alignment so that its anchor-plus-masks inputs produce one proposal distribution per input position. Comparing proposal lengths therefore requires translating each interface's block-size convention. [3, 4]

DSpark lets cheap token choices depend on one another

Even when parallel draft positions share context representations, a later position does not automatically know which token an earlier position actually sampled. DSpark illustrates this with two plausible phrases, “of course” and “no problem”: separate position distributions can combine into “of problem.” The issue is the dependence between realized choices, not whether each word looked plausible on its own. An early rejection then wastes the rest of the block. [4]

DSpark keeps the expensive backbone parallel and adds a lightweight left-to-right output stage. Its default Markov head takes the previously selected token and looks up a low-rank embedding. It projects that embedding into a transition bias: an adjustment to the score of each token in the vocabulary. Adding this bias to the current position's base logits gives corrected scores for sampling the next token and computing the proposal probability used by verification. After selecting “of,” the next position can favor “course.” [4]

This serial work is much smaller than rerunning the whole Transformer backbone. The default head models the immediately preceding token beyond the backbone's shared context; an alternative recurrent head carries more within-block history. The design question is how much additional surviving prefix repays the head's latency. A cheap serial head can improve coherence while the expensive backbone work remains parallel. [4]

DSpark budgets the target's work separately

DSpark chooses verification length separately from proposal generation: it admits a prefix of each completed draft to the target check. Trimming a suffix can save target computation, while the parallel backbone pass that produced it remains a cost of the round. [4]

A confidence head estimates the probability that each proposal passes, conditional on all preceding proposals surviving. Multiplying those conditional estimates gives each position's probability of belonging to the surviving prefix. A later token in a strong request can consequently be more useful than an early token in a doubtful request. Across active requests, the scheduler prices expected committed output against an engine-specific table of target steps per second at different verification batch sizes. That table is profiled at engine initialization; the scheduler uses it to change the allocation as load changes. This calculation needs calibrated probability magnitudes to estimate how much useful output each allocation will deliver. [4]

Admission also has a correctness constraint. If the scheduler looks at an already sampled token's downstream consequences and then decides whether to admit that token, it can favor particular token values. In an analytical counterexample, DSpark's Appendix A constructs a two-token vocabulary: the target wants (A, B) = (0.7, 0.3), while the proposal uses (0.5, 0.5). [4] A retrospective policy admits A because its continuation looks confident, but omits B and draws afresh from the target. The resulting probability of A is 0.5 + 0.5 × 0.7 = 0.85, not 0.7. [4] Biased admission has changed the candidates reaching the verifier and, through them, the output distribution. [4]

The paper consequently requires non-anticipating admission: whether a candidate enters verification must not depend on that candidate's realized value or later candidates. Its theoretical scheduler stops at the first non-improving extension. For production, the authors describe an asynchronous adaptation with two decisions: capacity determines how much verification work is allowed; ranking determines which current candidates receive priority. They use predictions from two steps earlier to choose capacity, and current scores to rank candidates. Their argument for preserving the target distribution rests on this historical capacity decision isolating admission from the current token's realization. [4]

For savings to reach the hardware, the serving engine must actually reduce verification work. Padding every shortened request back to the original width can spend the slots the scheduler intended to save. Cost profiles also depend on the engine and context regime: vLLM's October 2026 adaptive-verification documentation describes shape profiling, backend restrictions, and a default synthetic 8192-token context that may need adjustment for long-context deployments. [7] Useful confidence is only one input; useful execution shapes are another. [7]

DLoop buys more drafting before paying for a target check

DSpark asks how much of a proposal deserves checking. DLoop asks whether a confident drafter should produce another block before checking at all. After each complete drafting stage, DLoop sums the log probabilities the drafter assigned to that stage's proposed tokens. If the score passes a fixed threshold, it performs another stage; otherwise, the target checks all accumulated proposals together, including the block that stopped the loop. An early rejection discards every later block. [5]

Continuing creates an input problem for a target-feature-conditioned drafter. The target has not yet processed the new proposals, so it has no hidden representations for them. DLoop fills those positions with the drafter's own final-layer hidden states, and uses the preceding stage's last proposed token as the next anchor. These are provisional inputs for further drafting, not accepted output. Target verification still determines which tokens survive. [5]

Loop-aware training prepares the drafter for that longer chain of draft-produced representations. It unrolls extra drafting stages and supervises the first and last stages. The procedure reproduces the hidden-state chain alongside ground-truth training tokens. Parameter count stays unchanged, but extra draft passes consume training time and memory, as well as inference time when the loop continues. [5]

Postponing checks also gives up intermediate bonus opportunities. With S stages of k proposals, one final check can produce at most S k + 1 tokens; S individually successful checks could produce S(k + 1). DLoop exchanges those bonus opportunities for avoided target passes. Its confidence gate is cheap to evaluate, but every additional drafting stage must earn its cost. [5]

In the inspected released benchmark, DLoop's non-tree DSpark branch calls head_generate; the tree branch dispatches elsewhere. That driver samples through the Markov head, gates extra stages using corrected proposal logits, and submits the accumulated sequence to one greedy target check. It does not call the confidence predictor, the step-rate cost table, or load-aware prefix selection. This non-tree path combines dependency-head sampling and extension, while confidence-based global prefix allocation remains a separate operation. [8, 9]

A nearby name denotes a different operation. D-Loop, with a hyphen, retains an initial prefix from a proposed block and runs the same trained drafter again to regenerate that block's suffix with the prefix revealed; its method can also consider multiple prefix boundaries. D-Loop refines within a block. DLoop extends across whole drafting stages before verification. Their extra passes change different parts of the continuation. [5, 10]

Measure elapsed time per committed token

Extending the original cost analyses to a decoding interval gives this accounting identity:

total elapsed decode time / total newly committed tokens

Count accepted proposals and target correction or bonus tokens in the denominator. Discarded proposals consumed work but produced no committed output. For non-overlapping phases, round time includes drafting, verification, and overhead; when phases overlap, measure the elapsed critical path instead of adding concurrent work twice. Across unequal rounds, dividing total time by total tokens weights each round by the output it actually delivers. [1, 3]

For hypothetical arithmetic, suppose ordinary target decoding costs 10 milliseconds per token. A speculative round taking 14 milliseconds and committing four tokens costs 3.5 milliseconds per token, about 2.9 times faster. The same round committing only one token costs 14 milliseconds per token and is slower. The successful round amortizes its extra latency over several delivered tokens.

Why can doing extra calculations save time? With small batches and short blocks, a large model can spend much of a step moving weights and cached context through memory. Several candidate positions can share those reads while using arithmetic capacity that was otherwise underused. Chen and colleagues also describe small distributed communication operations that remain latency-bound as the short block grows. With enough available resources, these conditions can make checking several positions only modestly slower than a one-token step. Verification cost changes as block width and load consume that capacity. [1, 2]

When many requests already occupy the machine, speculative positions compete with productive positions from other requests. Longer contexts, model depth, kernel shapes, padding, and synchronization can move the break-even point too. That explains why DSpark may prune a suffix while DLoop may extend a draft: the former avoids target work, while the latter spends draft work to amortize a target call. The useful choice is the one that buys more surviving output for its elapsed cost.

Read a speedup together with its denominator

The named papers report results in different regimes. Each row below pairs an author-reported measurement with its own baseline and execution conditions:

The matched ablations reveal more than their largest multiplier. In DLoop's Table 2, looping a Gemma4-12B multi-token predictor without loop-aware training raises mean acceptance length from 4.86 to 6.06 while lowering speedup from 3.17× to 2.90×. [5] The extra accepted output does not repay the extra work in that configuration. Adding the training changes the result to 7.50 accepted length and 3.68× speedup. [5]

Load can reverse an incremental gain even within the same study. DLoop's SGLang/Qwen3-8B MATH-500 comparison reports: [5]

These percentages are calculated from Table A2's reported throughput. The study's general setup is H100/bf16/greedy; the appendix does not separately enumerate every serving flag or request-length distribution. These are incremental changes against DFlash; at concurrency 32, both configurations remain above the autoregressive baseline of 3763 tok/s. [5] In the same table, adding DLoop to DSpark at concurrency 32 instead improves MATH-500 throughput by about 4.5%. The operation's value changes with its base drafter as well as load. [5]

Finally, decoding cadence is only part of waiting for an answer. Aggregate tok/s measures system output rate; tok/s/user describes an individual generation stream. Time to first token and request-arrival-to-completion latency include additional stages. One concrete timing boundary appears in the inspected DLoop head-based evaluator: it records target prefill separately and resets its decode timer after the initial draft backbone pass. Its reported time per output token therefore measures the subsequent decode interval, excluding that startup work. [9]

When reading a speedup claim, identify the target, baseline, sampling rule, hardware, engine, context/output lengths, serving load, and timed boundary. Then ask whether the improvement comes from cheaper proposals, a longer surviving prefix, less verification, or fewer target rounds. Those distinctions explain what might transfer to another workload—and what must be measured again.

Conclusion

A draft accelerates generation by making candidate histories available before the target has chosen them. The target can score those histories together, while prefix-respecting acceptance and appropriate correction determine which computations become usable output. This separates the order in which predictions are evaluated from the order in which text is committed.

DFlash, DSpark, and DLoop exploit different parts of that separation: reuse target context for inexpensive parallel proposals, introduce cheap dependence and allocate verification slots, or continue drafting before another target call. Their gains depend on the same underlying balance: how much continuation survives for the elapsed work spent. A longer or more accurate draft is valuable when it improves that balance. The next practical question is how to choose draft extent and verification budget together under the actual engine's load, context, and execution costs.

Limitations

This account uses primary papers and read-only implementation captures available through October 31, 2026. The results are author-reported; no models or retrieved code were executed. The inspected evidence does not establish a complete nonzero-temperature verifier audit for every backend or a matched production evaluation combining DLoop stages with DSpark's global confidence scheduler. Some table-specific hardware, precision, length, and serving details remain unspecified, as noted alongside the results. The captures also do not quantify how much queueing and prompt processing contribute to whole-request latency across deployments.