How to Full-Train a Distilled Reasoning Model Without Making It Worse

The clean trace is not the transferable object

A small model can imitate a beautiful solution and still become a worse reasoner. In the clearest low-data example in this evidence pool, code performance fell below the starting checkpoint before recovering as the trace set grew. Easier questions helped more than harder ones in the measured low-to-medium-data regime, while the effect of correct versus incorrect teacher responses was small after the question distribution had changed.[1] The wrong lesson would be to train incorrect solutions. The useful lesson is that a correct final answer is only one property of a training example.

The endpoint can pass while the path is ungrounded, redundant, or too expensive for the student to reproduce. A full-parameter software-agent study made that distinction concrete: curating intermediate progress against a privileged process graph beat outcome-only filtering under size-matched and compute-matched controls, while also producing shorter trajectories.[2] Correctness is necessary for a positive demonstration, but it does not measure coverage, process validity, adaptation cost, or whether the student can represent the decisions inside the trace.

That changes the governing model of distillation. The job is not to rank teacher strings on a single quality axis. It is to construct a learnable decision process around the student's current policy, capacity, deployment budget, and capabilities that must survive the update. Student-native data, style-aligned repair, and selective feedback all support that model, but broad static SFT is an important counterexample: student-centered design does not imply that policy learning must follow every SFT run.

This article uses full-parameter post-training in a narrow sense: start from an existing pretrained or instruction-tuned student and update all of its parameters on reasoning-task data. Training from scratch, continued pretraining on raw text, LoRA, QLoRA, and other PEFT regimes are outside the recipe. Their results can suggest a mechanism, expose a diagnostic, or mark a limitation; they cannot establish what to do in an all-parameter run.

The first artifact is the frozen acceptance envelope

Before synthesizing a trace, freeze the experiment the final checkpoint must beat. Record the exact starting revision, tokenizer, chat template, EOS token, maximum training and generation lengths, loss mask, precision, optimizer, decoding settings, evaluator version, and number of samples per problem. Save a decoded batch and a zero-update evaluation. Log the number and names of trainable parameters at runtime; a full checkpoint or ZeRO sharding does not prove that every parameter receives gradients.

Choose the starting checkpoint by running the deployment tasks, not by assuming that the smallest nominal parameter count is the right student. Check whether it can parse the domain, use the required language, finish inside the context budget, follow the intended answer format, and solve a nonzero share of easier tasks. Base and instruct checkpoints should be a matched comparison when both are plausible. Research does not supply a universal checkpoint-selection rule, so checkpoint compatibility is a pre-training diagnostic, not a finding that one family or size must be used.

Freeze five evaluation views before any tuning:

1. a contamination-checked in-domain set; 2. an out-of-domain reasoning set; 3. repeated sampling for pass@1 and pass@k—success within k sampled attempts—or Avg@k, with the response count fixed; 4. completion, termination, response-length, latency, and token-cost diagnostics; 5. a retained-capability suite covering the general instruction, factual, multilingual, safety, and short-answer behavior that matters for deployment.

These controls catch failures that a headline accuracy hides. Open-R1's model card uses different response counts by benchmark and warns that the 30-problem AIME set has high run-to-run variance.[3] Light-R1 observed deviations above three points with sixteen or fewer responses and also recorded a GPQA regression during math-focused post-training.[4] LiveCodeBench attaches release dates to problems so evaluation can be restricted to items after a model's cutoff.[5] The pre-training evaluation contract is therefore part of the training recipe, not a report assembled after a good number appears.

That envelope changes how the candidate pool is built

Build problems before selecting traces. Start from provenance-tracked sources, remove benchmark overlaps and near-duplicates, and preserve a newly generated held-out slice. Generate multiple candidates per problem across plausible teachers, prompts, strategies, and lengths. Store the problem identity, source, teacher, prompt, seed, verifier result, trace length, and difficulty bucket so later filtering cannot silently change the population being compared.

The broad pool must come first. One data program tested question sources, mixtures, filtering, deduplication, multiple answers, and teacher choice across a common evaluation pipeline; its candidate construction made those choices separable rather than folding them into one opaque “quality” score.[6] A separate code pipeline generated multiple solutions for a large, decontaminated competitive-programming pool and showed continued gains as coverage scaled.[7] Its correctness-filtering ablation also illustrates the trap: the correct-only and incorrect-only subsets differed in problem difficulty and coverage, so the downstream comparison could not identify correctness as the cause. Failed solutions remain unsuitable positive targets; the ablation shows why selection must hold problem identity and coverage fixed.

Selection begins only after this broad pool exists. Stratify by problem, domain, difficulty, teacher, length, and verifier outcome. For each stratum, estimate the untouched student's completion rate, likelihood of the trace, final-answer success, and truncation behavior. Those measures are not a universal utility score. They expose where a polished trace asks the student to cross a representation gap it cannot yet cross.

Verification must preserve the problems that failed

Outcome verification and process verification answer different questions. The outcome checker decides whether the final answer, program, proof object, or tool result passes. The process checker asks whether the intermediate states are valid and worth imitating. In code, that may include syntax, compilation, unit tests, and execution at intermediate checkpoints. In formal math it may use symbolic or executable subclaims. In software agents it can compare discovered facts and milestones with a leakage-controlled process graph. The full-parameter evidence for process-grounded curation is strong in software engineering.[2] It does not establish a reliable verifier for free-form natural-language reasoning.

For an auditable trial, quarantine failed or disputed trajectories: never place them in the positive cross-entropy set, but retain their problem identifiers and verifier records for later repair, contrastive preference data, or hard-negative evaluation. This quarantine rule prevents a failed rollout from disappearing along with a difficult problem. Its use outside executable or otherwise grounded domains is a control proposal, because Research did not establish a target-scope natural-language repair pipeline.

Judge-rated polish is not a substitute. In one experiment, oracle-refined traces received better reward scores but trained worse students than student-native rejection-sampled traces; preserving the student's style while repairing logic recovered utility.[8] This supports adaptation cost as an explanation for the gap between perceived quality and training utility; it does not isolate adaptation cost as the sole cause. A verifier should certify what it can actually observe, while student compatibility remains a separate selection variable.

Trace length becomes a controlled variable

Once outcomes and processes have separate records, build trace-format arms from the same accepted problems. Keep the raw verified trace as the accuracy control. Add a structure-preserving rewrite that removes repetition without dropping decisive computations. Add an answer-only arm only as a diagnostic. Match problem identities and, where possible, optimization steps or trainable-token budgets.

The strongest explicit full-parameter grid in the pool does not support universal compression. Across its teachers, students, and update methods, raw traces were most accurate; model-compressed traces reduced training and inference cost and usually beat naive truncation at comparable shortened length.[9] Answer-only full-parameter runs were often unstable or failed to terminate. A narrower long/short mixture study reported a better accuracy–length trade-off, but its captured update scope was not explicit.[10] The defensible procedure is therefore to choose a point on an accuracy–efficiency frontier, with raw traces retained as the comparator.

Capacity still matters. A study spanning four teachers, seven students, and seven reasoning datasets found non-monotonic trace-granularity effects: weaker students often preferred simpler supervision, while stronger students could use finer reasoning, and the strongest teacher did not always produce the strongest student.[11] That evidence makes capacity and style useful diagnostics. It does not overturn the target-scope raw-trace result or justify an automatic “shorter for smaller” rule.

Difficulty must be treated the same way. Scale broad problem coverage and plot learning curves by difficulty bucket. Do not hard-code easy-first progression: some data studies favor difficult, disagreement-rich, long, or strategy-diverse examples, while the code valley favored easier examples at its measured stages. A competence-adaptive curriculum is a reasonable experiment—advance a bucket when completion and held-out accuracy improve—but the pool does not establish it as a closed-loop full-parameter recipe.

Carry the selected raw or structure-preserving trace arm, unchanged, into the conservative all-parameter SFT control. That gives every later curriculum or policy-learning branch a common data representation to beat.

The conservative SFT run is the decision point

The first training run should be deliberately ordinary: all parameters train, the response uses cross-entropy, the learning rate is swept conservatively, the template and EOS match inference, the loss excludes prompt tokens unless a separate arm tests otherwise, and checkpoints are evaluated throughout the learning curve. Keep packing off until attention masks and sample boundaries have been inspected. Stop on the frozen target, completion, and retention gates rather than on training loss.

Direct SFT can be the endpoint. DeepSeek trained six smaller Qwen and Llama students on roughly 800,000 curated samples and applied no RL stage to those distilled students; its distilled 32B checkpoint also outperformed a same-size model trained for more than 10,000 RL steps.[12] Open-R1 supplies an independent open SFT reproduction, and OpenCodeReasoning supplies a large-scale code case.[3, 7] These are bounded successes in math and code, not proof that any verified trace set will work. They do establish that RL is not a prerequisite.

The loss experiment starts from that positive control. Use low-learning-rate response cross-entropy (CE) as the baseline, then add one intervention at a time while keeping data, steps, batch construction, and evaluation fixed. A full-parameter domain-SFT study found that a smaller learning rate could preserve comparable target performance with much less general degradation, although it did not eliminate degradation in every setting.[13]

Keep four possible signals separable:

• answer emphasis, which changes how much the final answer contributes relative to rationale tokens;
• process masking or weighting, which restricts positive loss to steps the process checker can support;
• selective teacher or preference guidance, which changes supervision at identified decision tokens;
• retention data or a reference constraint, which limits drift on non-target prompts.

No target-scope source jointly tests those four terms. One study reports that vanilla CE can reduce entropy—the model's uncertainty or sampling diversity in this setting—and that blanket entropy maximization is not a cure, while another reduces uncertainty selectively at tokens where the student is uncertain and the teacher is confident.[14, 15] The studies optimize different handoffs, which helps explain why the directions conflict. The proposed multi-part loss therefore remains a set of single-factor tests, not an established objective, and no combined formula or universal set of weights belongs in the recipe.

Those four arms modify the supervised objective or the pressure to retain prior behavior. The branches below instead change where training examples come from or how student rollouts update the policy.

Three branches begin only after that checkpoint

The frozen SFT checkpoint now creates a real decision. If it meets target, completion, efficiency, and retention gates, stop. If it fails, identify which signal is missing before choosing a branch. Dense on-policy distillation, rejection-sampling SFT, and reinforcement learning with verifiable rewards (RLVR) consume student rollouts in different ways. Treating them as one generic “RL stage” would hide the reason each might help.

Dense guidance tests student-visited prefixes.

Teacher-forced SFT trains on prefixes selected by the teacher. One early student error changes the prefixes reached at inference, so later predictions occur in states absent from the static trace. Dense on-policy distillation samples from the student and evaluates teacher distributions or preferences at those student-visited prefixes. That directly tests the exposure-bias mechanism.

Enter this arm only when the student already produces usable, nondegenerate rollouts. Keep prompts fixed against the off-policy and SFT controls. Prefer selective or locally supported guidance, which limits teacher influence to tokens the teacher can score reliably: one on-policy variant beat its off-policy counterpart across the evaluated benchmarks, while its component tests favored selective and indirect feedback over forcing a teacher distribution at every token.[15] But “on-policy” is only a statement about where prefixes came from. Sampled-token guidance can become unreliable after prefix drift or special-token mismatch.[17]

This is an experimental arm, not a universal continuation. Bound teacher influence to tokens or local support the teacher can score reliably. Track completion, length, entropy, held-out accuracy, and drift from the SFT checkpoint. Stop when the teacher becomes unreliable on student states, when repetition grows, or when the matched SFT control wins.

Rejection-sampling SFT recycles complete student successes.

Rejection-sampling SFT uses a coarser signal. Sample several complete trajectories from the current student, verify the outcomes, and apply cross-entropy only to accepted student-native positives. Resample after each round so the data distribution moves with the student. Balance by problem identity and pass-rate bucket; otherwise easy prompts contribute many positives while hard prompts vanish. Preserve rejected problems in the quarantine pool for repair or preference experiments, outside the positive CE set.

Matched post-training studies support this branch when the checkpoint already produces enough correct and diverse trajectories, but the captured artifacts do not fully audit all-parameter updates for either study. In one comparison, positive-only training learned quickly, then lost entropy and plateaued; Group Relative Policy Optimization (GRPO), an RLVR method, finished slightly higher under the same four-rollout setup.[18] A revision-based on-policy study outperformed rejection fine-tuning, GRPO, and self-distillation fine-tuning under the same question set and training-sample budget.[16] Do not enter when pass rates are near zero or saturated near one, and stop if positive-only training collapses pass@k or strategy diversity.

RLVR spends negative feedback only when it is informative.

RLVR updates the current policy from scalar rewards rather than turning accepted trajectories into a static CE set. It earns the extra complexity when the verifier is trustworthy, batches contain a useful mix of successes and failures, and the SFT or rejection-sampling checkpoint still has exploration headroom. Compare at matched questions, rollout tokens, wall-clock budget, and stopping points.

The boundary is empirical. In one study, prompts whose sampled responses were all wrong were actively harmful, and filtering those prompts mattered more than reward normalization.[18] In another, the matched GRPO baseline lost to revision plus distillation, and extra rollouts did not reliably close the gap.[16] A separate SFT-distilled 32B result further warns that small-model RL can spend large compute without beating teacher transfer.[12]

Apply RLVR only where outcome rewards are difficult to game and remain nonsaturated. Preserve an untouched SFT and rejection-sampling checkpoint. Stop when reward variance vanishes, pass@k or entropy contracts, held-out results stop improving, or the retention floor is crossed. RLVR can improve sampling efficiency or explore beyond the accepted set; this pool does not show that it can reliably import reasoning knowledge absent from the starting policy.

Retention gates decide which checkpoint survives

Full-parameter training can move every capability encoded in the checkpoint. A lower target loss may coexist with worse instruction following, factual recall, concise answering, multilingual behavior, or unrelated reasoning. Measure those axes at every saved checkpoint and declare rejection thresholds before training.

Start with the least elaborate protection: conservative learning rate, frequent checkpoints, and the frozen retention suite. Add replay only after measured drift. When testing replay, compare no replay, distribution-matched replay, and—separately—a reference-policy penalty that limits movement away from the starting policy. Continual full-parameter SFT evidence shows that high-score on-policy replay can reduce forgetting, while low-score replay from the same rollout pool can be worse than ordinary replay.[19] Domain-adaptation evidence likewise suggests that rehearsal works best when it resembles the starting model's useful instruction distribution.[20]

These studies are adjacent to one-shot reasoning-trace distillation. They justify retention controls as diagnostics and ablations, not a mandatory replay-plus-KL bundle. A reference constraint can preserve old behavior yet prevent acquisition when the target requires a large policy shift. Reject any checkpoint that buys its target gain by crossing a predeclared retention floor.

The acceptance envelope decides whether a checkpoint survives; matched causal checks explain why it moved. Run the following checks at reduced scale, changing one factor while holding problem identities, update scope, optimizer budget, template, decoding, and evaluator fixed:

Plot target accuracy, out-of-domain accuracy, completion, length, pass@k, entropy or calibration, and the retention suite across checkpoints and data quantities. Report sampling distributions rather than one best run. A causal-check PASS does not make a checkpoint acceptable, and an envelope PASS does not identify a cause. A successful full train is the smallest intervention that clears all predeclared gates, not the checkpoint with the highest single benchmark score.

Limitations

The evidence is concentrated in mathematics, competitive code, Qwen-family students, and a narrow software-agent setting. Several valuable capacity, entropy, and on-policy studies do not make their all-parameter update scope fully auditable in the captured artifacts. OpenCodeReasoning's official training code was not captured. Natural-language process verification remains much weaker than executable checking. No study compares one strong full-parameter SFT checkpoint with dense on-policy distillation, rejection-sampling SFT, and RLVR under the same questions, rollout tokens, compute, verifier, stopping rule, and retention suite.

The archived raw PDF captures for the Quality-Utility Paradox [8] and CurioSFT [14] are byte-mangled and unreadable at the source-integrity threshold. Their cited claims are traceable to the extraction layer but cannot be checked line by line against the archived raw text, reducing auditability.

The cleanest retention interventions come from continual or domain adaptation rather than the exact one-shot reasoning-trace regime. No target-scope experiment validates the combined answer, process, selective-distillation, and retention objective. No universal rule establishes when a compressed trace, difficult example, or selective entropy direction is right for a particular student. Those gaps are why several stages above are controls or conditional arms rather than defaults.

The run ends with an open comparison

The root strategy is to make every stage answer to the student and to a frozen measurement contract. Build coverage before filtering. Verify endpoints without confusing them with process. Keep raw traces as the accuracy control. Let conservative full-parameter SFT prove whether it is already enough. Only then open one student-generated branch at a time, and keep a checkpoint only if its reasoning gain survives completion, efficiency, out-of-domain, sampling, and retention gates.

What remains open is the comparison practitioners most need: across model families and beyond math and code, does any student-generated branch reliably beat a strong, retention-safe full-parameter SFT baseline when every data, verifier, compute, and evaluation variable is matched?

Appendix: Implementation resources

These resources are starting points for running the matched comparison left open above; none replaces the runtime audit of an all-parameter update.

The most complete starting stack in the captured pool is Open-R1, its OpenR1-Distill-7B model card, TRL's SFTTrainer, and Lighteval. Open-R1 exposes the SFT launch and evaluation path.[21] The model card records lineage, optimizer, seed, decoding, per-benchmark response counts, and logs.[3] TRL documents adapter training as a separate configured path.[22] Together, these are the strongest captured starting point for an all-parameter run, but the exact YAML and trainer source were incomplete in this freeze. Pin the configuration, then log trainable-parameter counts and gradient coverage at runtime rather than inferring full-parameter scope from ZeRO-3.

For data and controlled SFT studies, use OpenCodeReasoning for large-scale code traces, contamination procedures, training hyperparameters, and filtering and scaling ablations.[7] Data Recipes for Reasoning Models supplies experiments on candidate source, mixture, deduplication, multiple answers, and teacher selection.[6] OpenCodeReasoning is a paper, dataset, and checkpoint resource here, not a captured drop-in all-parameter training implementation.

For dense-guidance trials, the official TSD-KD repository exposes its Qwen2.5-1.5B-Instruct start, prompt data, distributed launch, code, checkpoint, and core loss hyperparameters.[24] Its main-study all-parameter scope remains incompletely auditable from the captured configuration.

For revision-based on-policy trials, the SD-Zero paper supplies a matched comparison with rejection fine-tuning and GRPO; its repository page was not captured in this freeze.[16] For capacity and process diagnostics, Distilling-CoT-Reasoning covers granularity and teacher capacity,[11, 25] while Patches-to-Trajectories supplies the software-domain reference for executable intermediate artifacts.[2] Implementations that explicitly use LoRA or PEFT remain diagnostic comparators only; they are not substitutes for the full-parameter recipe.

For evaluation, the LiveCodeBench repository provides date-windowed code evaluation across generation, repair, execution, and test-output scenarios.[27] Its paper documents the temporal-contamination rationale.[5] Pin the evaluator version and release window, then pair it with domain-specific held-out sets and the deployment retention suite; no captured public benchmark bundle supplies the complete acceptance envelope for this commission.