How a Reasoner Learns When to Stop

When the tenth turn destroys a correct answer

On SVR's GSM8K trajectories, a correct answer appeared at some point within ten turns for 0.925 of the examples.[1] Force the same policy to return its tenth answer, however, and accuracy fell to 0.736.[1] The extra turns found solutions that the trajectory later erased.

The loss is easiest to understand by following the answer as a state rather than counting the tokens used to produce it. Productive depth changes the answer for good reasons. Internal stagnation repeats a state before an answer. Post-answer drift changes a state that was already correct. Checkpoint studies have observed the balance between wrong-to-correct repairs and correct-to-wrong revisions tipping at different budgets for different open models and tasks; the first correct checkpoint is visible to an evaluator, not to the deployed model.[4, 9]

That leaves the inference system with a decision each time it has a usable answer: keep this state, or expose it to another operation. A budget is valuable when it preserves a choice among actions, including the choice to keep an earlier answer. If continuing is compulsory, more compute can deliver less even though it created a useful state along the way. The practical question is therefore not how many tokens make a model smarter. It is what the system can observe before the next action, and whether that observation is reliable enough to protect the answer already in hand.[1]

What the stop/refine machine can observe

At every SVR turn, the same policy emits three objects: a proposed solution, a discrete verdict—Correct, Incorrect, or Unsure—and a confidence score. The controller retains a complete answer only when the verdict is Correct and confidence clears a threshold; otherwise it supplies the problem, retained draft, and self-verification state to another refinement turn.[1] Verdict and confidence are learned signals. They are not the correctness oracle used to score the experiment.

Before those signals are trusted, the trajectory and its alternatives have to be measured without collapsing different failures into one accuracy number. Vary sequential turns, independent sample count, prefix-search depth or width, verifier calls, and stopping threshold in separate runs. Keep the prompt, decoder, reducer, verifier, stop rule, and budget attached to every result, because changing any one changes the inference system being evaluated.[8]

The resulting checks answer different questions:

For a declared candidate bank, oracle pass@N asks whether at least one candidate is correct. If coverage remains low, no selector can recover an answer from that bank. That result is conditional on the named prompt, sampler, tools, and budget; it is not proof that the base model lacks the capability under every proposal.[7, 8]

Once a bank does contain a correct answer, compare its oracle coverage with selected accuracy and inspect ranking, calibration, parsing, and harm to the first correct candidate. The gap proves selection loss.[7] Separate ablations are still needed to assign that loss to ranking, calibration, parsing, answer correlation, or domain shift. On the sequential trajectory, force answers at regular checkpoints and record first-correct time, wrong-to-correct repairs, and correct-to-wrong revisions. Define stopping regret per example as the utility of the best earlier checkpoint minus the utility at the model's chosen stop. Both the first-correct time and that best checkpoint are hindsight oracles, so they evaluate a gate without becoming inputs to it.[9, 23]

Together, these observations locate the state that training should improve. They do not safely decide which states an immature controller is allowed to see. If an unreliable early gate controlled data collection, confidently wrong states could terminate before the learner observed the repair, while correct states could disappear before the learner observed the damage. That is why the training trajectory runs past the point where a mature deployment would stop.

Why training runs beyond the desired stop

SVR starts from an existing post-trained Qwen3.5-2B reasoner and updates all of its parameters. During training, every rollout executes a fixed number of refinement turns regardless of the policy's verdict or confidence. Later states therefore remain in the learning problem even when the current controller would have hidden them.[1] Here, FULL TRAIN means full-parameter post-training of an existing reasoning model, not training from scratch, continued pretraining, PEFT, or an inference-only controller.

Each fixed-horizon trajectory carries two kinds of signal. The solution term rewards a correct current answer. Across turns, the transition term rewards a wrong-to-correct repair or the preservation of an already correct answer, while suppressing a correct-to-wrong regression or repeated failure. A separate verdict–confidence block calibrates confidence to correctness, suppresses confident false positives, and makes correct states ready to stop. Ground-truth checks construct these rewards but never enter the refinement prompt or inference controller. Removing the verification block, calibration term, false-positive control, stop-readiness reward, or half of the verdict–confidence interface worsened the corresponding control behavior in the reported ablations.[1]

The fixed horizon is not a deployment budget and the training objective does not price realized tokens, latency, branches, or verifier calls. It is a way to keep the learner from censoring its own counterexamples. For an existing post-trained reasoner on tasks with exact outcome checks, the directly supported stage is therefore: use full-parameter, fixed-horizon RLVR to train stop versus refine with correctness, correct-state preservation, calibration, false-positive control, and stop readiness.[1] Any explicit compute charge remains an ablation until it is shown not to reward a short wrong answer.

Only after later states have taught the verdict and confidence what repair and damage look like may those signals control which answer leaves the trajectory.

When verdict and confidence may end the trajectory

At inference, the schedule changes while the policy-visible information stays the same. A complete answer with a Correct verdict and confidence above the selected threshold is retained; otherwise the policy refines until the maximum turn budget is reached. The gate now acts on the evolving state that fixed-horizon training kept visible.[1]

The comparison changes only the retention rule. Using the same trained policy and greedy decoding, the best forced fixed-turn operating point reached 0.450 on the All-7 macro-average.[1] Adaptive SVR reached 0.563 while executing 2.99 turns on average.[1] The difference is not a claim that every additional turn is harmful. It shows that one shared turn count cannot keep the best state for every trajectory: one answer still needs repair while another is already at risk of being overwritten.

The learned signal is useful without being infallible. At SVR's default threshold, All-7 weights each evaluated benchmark equally, and premature-stop error measures the share of all evaluation examples that stopped early with an incorrect returned answer; that error remained 29.9%.[1] Lower thresholds can accept a confidently wrong state; higher thresholds can delay retention until after a correct state has been revised. The clearest full-parameter result is therefore bounded: a reasoner learned a stop/refine policy that beat its forced-turn alternatives under the reported conditions. It is not evidence for a universally reliable confidence threshold, an explicit realized-cost objective, or a controller over every compute action.

Once stopping can preserve an earlier answer, the budget produces a delivered accuracy–cost consequence rather than merely a longer trace. The next question is which part of that consequence survives when the optional action changes.

What the delivered frontier says about more compute

A fixed procedure can become less accurate as its budget grows because it is compelled to take an action whose expected corrective value has fallen below its latency, selection, or revision risk. Across eight open models and four reasoning datasets, fixed beam policies sometimes flattened or declined as their generated-token budgets grew, while the best observed strategy available at each budget formed a monotone envelope for each model family.[2] The envelope can reuse a lower-budget strategy. A fixed arm cannot.

The same distinction appears when allocation happens before generation rather than after a refinement turn. A learned allocator chose sample budgets per input and beat uniform allocation under matched average budgets on the studied MATH and GSM8K settings.[3] Its learned component is a lightweight classifier that predicts oracle budget actions from input features, making it an allocation comparator around SVR rather than an update to the reasoner's parameters.[3]

Parallel sampling also exposes a different bottleneck. Adding candidates cannot reduce oracle coverage for a fixed bank, but the delivered answer can stall or fall when the bank is homogeneous or the reducer is harmful. Across three seeds for Qwen3.6-35B-A3B-BF16 with five samples per GPQA-Diamond problem, 87.54% of candidate banks were answer-identical and the oracle improvement space was 3.03 percentage points.[7] The measurements show little recoverable mass in those banks; they do not establish a causal law that samples become correlated as their number grows. A set-aware external verifier improved calibration, best-of-N selection, and streaming early stopping in a separate parallel math setting, under assumptions of parallel resources and low verifier overhead.[5]

Prefix search puts the risk earlier. In a 23-problem case study using one Qwen2.5-Math generator–PRM pair, PRM-guided tree search did not significantly beat Best-of-N despite higher cost, and its process value became less reliable with depth and out of distribution.[6] The result shows that repeated pruning can amplify a weak value estimate. It does not show that deeper search is generally harmful.

Stop versus refine remains the directly supported control. A failed sampling, search, verification, or allocation diagnostic admits only its matching branch around that control, and none of those branches inherits SVR's full-parameter scope. The next step is to freeze the control before changing one intervention.

How to turn diagnosis into a full-parameter run

To make each one-change comparison auditable, before the first update freeze the base checkpoint's refinement trajectories, candidate banks, verifier outputs, costs, splits, trainable-parameter count, and retention suite. Reproduce each mechanism-separated sweep on that checkpoint. The fixed-horizon full-parameter SVR objective remains the control; every other stage enters only when the matching diagnostic fails, carries a support metric, and has a condition for removing it.

Among these interventions, only stop/refine is directly supported by a full-parameter training study. Dynamic prompt selection also has full-parameter math RLVR support: it outperformed static selection and full-data training while pre-collected off-policy trajectories supplied its influence estimates.[11] TOPS reports shortest-correct selection and masking of identified erroneous rounds.[12] Its repository supplies the corresponding training and evaluation assets.[19] BudgetThinker and LAPO study explicit or problem-local effort levels.[13, 14] Those papers do not clearly establish that the solver itself was updated end to end, so they do not establish full-parameter adaptive SFT. Selective self-rewriting can test trace control, but the cited evidence does not show that a rewritten teacher trace matches the learner's executable reasoning capacity.[24]

Use the policy-refresh branch only when its diagnostic fails: compare current-policy refresh with stale-rollout and off-policy-influence controls. A full-parameter safety-alignment study favors current-policy supervision over fixed supervision.[15] Separately, full-parameter math RLVR uses pre-collected off-policy trajectories for influence-guided selection.[11] On-Policy SFT trains on self-generated responses filtered for correctness and concision.[16] LAPO uses problem-local length guidance.[14] If using rejection sampling, select within problem and budget cells and preserve diversity; do not claim that every trace must be freshly on-policy.

Calibrate on the states the controller can act on, report Brier score and confident false positives, and keep external pruning or set-level extensions as controls. Retention is measured rather than assumed: full-parameter reinforcement fine-tuning retained broad capability without replay in one multimodal continual-post-training study, while a reasoning-focused study with unclear update scope benefited from replay.[17, 18] Gate retention at every checkpoint. Add replay when regression appears; do not prescribe a fixed replay ratio or claim replay is always necessary.

A failed measurement names one branch; that branch must move its support metric without tripping its kill criterion. The complete trained inference system, rather than its best maximum-budget score, then faces promotion.

What a checkpoint must prove before promotion

Re-run the frozen sweeps after each parameter update and plot the delivered system, not an oracle-selected trace. Plot delivered accuracy against each cost axis, not just maximum-budget accuracy or average response length. Keep tokens, estimated FLOPs, wall-clock latency, peak memory, and verifier calls separate: theoretical compute savings need not become measured latency savings.[2, 10]

Slice the frontier by difficulty and domain, then apply run-specific thresholds to the delivered system. It must improve or preserve the delivered accuracy–compute curve, avoid low-budget regression, retain broad capabilities, remain calibrated under domain shift, pass safety checks, and meet realized tail-latency limits. The evidence supports these axes separately, not one universal joint cutoff.

Implementation artifacts shorten setup but do not waive that proof. TOPS provides training and evaluation commands, released models, data, and exact-answer tooling, but its public materials do not clearly state whether the solver itself is updated end to end.[19] LAPO provides a two-stage OpenRLHF pipeline and length-guidance wiring with the same ambiguity.[20] Verl provides rollout-correction controls for policy mismatch.[21] The DeepSeek guide provides ordinary full-parameter SFT infrastructure, not adaptive-budget learning.[22] Before importing any component, record frozen modules, trainable parameters, optimizer state, data provenance, policy lag, verifier, accounting, and splits.

Passing this gate would validate only the full-parameter run and delivered inference system that were tested.

What this machine does not settle

The strongest direct evidence remains one full-parameter stop/refine study on a post-trained 2B reasoner, concentrated in mathematics and other tasks with exact or crisp checks.[1] Before generalizing it, reproduce the result independently on the target checkpoint and tasks. The parallel-verifier result assumes parallel resources and low overhead; the search negative control covers 23 problems and one generator–PRM pair; and the cited studies do not jointly match tokens, FLOPs, latency, memory traffic, and verifier calls.[5, 6]

The target boundary is full-parameter post-training of an existing pretrained or instruction-tuned reasoner. LoRA, adapters, frozen-backbone test-time training, inference-only routers, external verifiers, and update-scope-unclear methods can motivate a mechanism, diagnostic, or negative control. They cannot justify a full-parameter training instruction. Continued pretraining and training from scratch are different interventions again. Filtering, rewriting, multi-budget SFT, local rejection, and external verification therefore remain adjacent or scope-unclear until reproduced under FULL TRAIN.

The cited induced-fault experiments do not jointly validate the diagnostic stack across base-policy, search, verifier, and stopping defects. No result cited here validates capacity-aware rewriting in this full-parameter setup. The two retention studies report different operational value for replay across different regimes, so replay remains conditional. Taken together, these results do not yet validate an end-to-end full-parameter adaptive-compute pipeline across math, code, factual, and open-ended reasoning.

Even a promoted SVR-style checkpoint would therefore answer only one choice facing the current answer: retain it or refine it. The missing experiment begins from that same state and expands the action set.

The controller that still has to be trained

The current answer may be kept, extended, replaced by another sample, split into branches, sent to a verifier or tool, or returned. A full-parameter experiment would have to expose those actions at measured realized costs while preserving the reachability of correct solutions and the capabilities the base checkpoint already had. It would compare each new branch against the fixed-horizon stop/refine control, then test the learned mixture across math, code, factual, and open-ended tasks on the same accuracy–cost accounting.

The open question is whether one calibrated full-parameter policy can make those heterogeneous choices without narrowing reachable solutions, accepting confident errors, or forgetting capabilities. Until that cross-domain frontier is measured and passes the promotion gate, the evidence supports a stop/refine core and a falsifiable program of interventions—not a finished all-action controller.