Can This Checkpoint Still Learn?

Present accuracy cannot answer whether to keep the checkpoint

A checkpoint can still score well on its current validation set while being a poor starting point for the next update. The current score describes what its weights already do. It does not measure how quickly those weights can move toward a new target under a fixed data and update budget.

That second property is usually called plasticity. In a repeatedly trained system, loss of plasticity is a history-dependent shortfall on a new-target learning curve relative to an appropriate fresh control. Catastrophic forgetting is different: it is a fall in performance on previously learned targets after an update. One model can forget without becoming hard to train, or become hard to train without the experiment ever measuring what it retained. The two outcomes therefore require separate scores.

Long-horizon experiments show why the distinction matters. In Continual ImageNet, conventional networks learned early binary tasks well but declined toward or below a linear baseline after many task changes. In class-incremental CIFAR-100, the same study kept old classes in the training stream, evaluated all classes seen so far, and compared each incremental model with a network retrained from scratch on the same class subset. The incremental model moved from an early advantage to a later deficit. These are controlled results from particular convolutional and residual networks, training schedules, and task constructions; they do not establish that neural networks inevitably age this way.[1]

They also do not prove the stronger claim that one checkpoint can hold old-target performance fixed while losing a fully parity-controlled race against a fresh model on a new target. The studies establish later-task learning deficits and the need to measure retention separately. The complete conjunction remains unverified. That boundary changes the operational question from “is this checkpoint still accurate?” to a comparison among three actions: continue it unchanged, reset its optimizer and training state before continuing, or retrain from scratch.

So the experiment needs two scores and three arms

The comparison begins when representative new data arrives, before any arm is allowed to become the production model. Save the aged checkpoint and create disposable clones. The test needs five inputs:

1. the aged model weights and a snapshot of every mutable training-state component, including optimizer moments, schedules, gradient-scaling state, and running statistics; 2. a fresh initialization of the same architecture and capacity; 3. a versioned new-target training set and untouched new-target evaluation set; 4. a frozen retention set covering the behavior that must survive the update; and 5. a predeclared budget for examples, optimizer updates, tuning trials, random seeds, wall-clock or accelerator use, and evaluation frequency.

Run three arms from those inputs:

The schedule-state convention follows the arm definitions: A preserves its existing schedule progress, while B and C restart schedule progress at the comparison boundary and use the same schedule configuration. All three arms receive the same allowed training data in the same order or from the same sampling distribution. They get the same number of examples, parameter updates, evaluation points, seed budget, early-stopping rule, and hyperparameter-search budget. If each arm is tuned, tune each over the same declared space and number of trials; if tuning is unaffordable, use the same precommitted settings and call the result a fixed-configuration comparison. Changing the data mixture, model width, or search effort for the scratch arm would turn a checkpoint test into a comparison of different training programs.

For future learnability, record training loss or accuracy and untouched new-target performance at every evaluation point. Summarize both the area under the learning curve and performance at the fixed budget. The area distinguishes a model that learns promptly from one that reaches the same endpoint only after consuming more updates. A fixed-duration per-task “average online accuracy” has been used for this purpose in continual-learning experiments, but that work explicitly left forgetting aside.[2]

For retention, score the frozen old-target set before the probe and again after the same number of updates in every arm. Report the pre-to-post change separately from the new-target curve. The scratch arm may begin with little retained capability; that is a real deployment cost, not a reason to remove the arm. If there is no old-target evaluation set, the experiment cannot distinguish plasticity loss from forgetting and should not promote an updated checkpoint on this evidence.

The three curves separate the first decision branches

Once the arms finish, their relationship matters more than any single endpoint. The first reading should use functional results, not weight statistics:

When continue (A) and reset-state (B) both lag scratch (C) on the new-target curve while old-target retention remains acceptable, the aged weights are only a candidate source of history-dependent loss: several rivals can produce that relationship. Repeat the comparison with a longer update budget to test ordinary slow convergence. Hold target difficulty fixed or include multiple calibrated canaries. Compare a capacity-matched wider control only as a separate factorial test, and run an optimizer-state reset before attributing the gap to the weights. Where a sequential-versus-joint comparison is used, account for co-observation. For example, a model trained on old and new samples together may generalize better because it saw both at once—not because the sequential model forgot or lost the ability to learn.

Internal measurements help choose among those rival tests. Weight and gradient norms, activation or dormancy fractions, feature rank, curvature, churn, and spectral condition can all be logged. They cannot replace the three learning curves. Causal studies identify several mechanisms in bounded settings, including preactivation shifts, parameter growth and sharpness, target-scale effects, and changes in curvature.[3] A direct counterexample shows why a universal alarm is unsafe: adaptive growing networks maintained plasticity while their fraction of dead units continued to rise.[4] An internal metric becomes an early warning only after it predicts a later functional canary in the same architecture, optimizer, transition, and horizon with useful lead time.

Data access determines which branch is deployable

The same curves can support different actions depending on what data may be used after the test.

If old and new training data are both available, define one joint or replay mixture before launching the arms and keep it fixed across A, B, and C. Scratch is then a genuine deployment candidate: it can be judged on both new-target learning and retained behavior using the same information as the aged arms. Replay may improve plasticity, retention, and co-observation together, so a win does not identify which mechanism was repaired; it only identifies a better training program under that data policy.

If only new training data are available but an old-target evaluation set remains, scratch is primarily a calibration arm. It can reveal that the aged weights learn the new target poorly, yet fail the retention floor because the information needed to rebuild old behavior is unavailable. In that branch, the experiment may justify a state reset or a bounded weight-level repair, but it cannot make scratch deployable without accepting the retention loss or recovering old training signal. The Nature experiments make the tradeoff concrete: full reinitialization is practical when past experience is retained in a large buffer, whereas in small- or no-buffer reinforcement learning it returns the agent toward its starting behavior.[1]

If neither old training data nor a trustworthy old-target evaluation set exists, retention is unknowable. The checkpoint may be preserved for operational reasons, but the experiment cannot support the claim that it “still remembers.” This is a stopping condition, not a missing column to fill with present validation accuracy.

When a system can deliberately schedule a change by controlling intermediate inputs, labels, or a mix of old and new samples, transition control opens a conditional branch: input or label interpolation and a gradually changing mix of old and new samples are testable arms. In synthetic random-label, pixel-permutation, and cipher tasks, gradual constructions reduced the observed loss relative to abrupt changes.[5] Those experiments do not show that naturally gradual production drift is safe, and smoothing is not an option when the new distribution arrives without controllable intermediate states. Together, data availability and transition control delimit the interventions worth testing after diagnosis.

Interventions enter only after a branch is identified

A state reset is cheap enough to include in every checkpoint comparison because it isolates a common rival explanation. Weight-level interventions deserve a narrower admission rule: test them only after A and B both lag C, and only in a regime that resembles the supporting experiment.

Regime match narrows the candidates. When both aged-weight arms lag scratch across a long sequence of supervised MLP or CNN tasks, the supported candidates include weight decay toward zero or initialization, shrink-and-perturb, and selective unit or weight reinitialization. A 2026 architecture comparison found selective weight reinitialization stronger than unit reinitialization in small or layer-normalized networks, but similar in sufficiently large networks without layer normalization.[6] Both variants therefore belong in the local ablation; neither replaces Arm B universally.

On-policy training needs a different test. A NeurIPS 2024 comparison found that several remedies transferred from supervised or off-policy settings were inconsistent and sometimes worse than no intervention, while regenerative methods helped in its gridworld, ProcGen, and Montezuma’s Revenge experiments.[7] A mismatch in optimizer, data collection, or environment transition rejects the transfer. The stationary expanding-data branch differs again: when training fit remains intact while held-out performance lags, DASH's warm-versus-cold experiments attribute the tested failure to noise memorization.[8] That relationship selects data- and noise-focused controls before a weight repair; it does not establish the same mechanism for every aged-versus-scratch gap.

The author-controlled shibhansh/loss-of-plasticity repository provides code for Continual ImageNet, class-incremental CIFAR, slowly changing regression, PPO, and continual backpropagation.[9] Use it as the bounded baseline when one of those experiment families matches the local failed regime, then add the three-arm canary and retention ledger before comparing an intervention with scratch. The repository does not provide a pinned environment, released checkpoints, or one runnable command for the complete two-axis comparison.

That missing pin, checkpoint set, and runnable command matter more than recency. The peer-reviewed anchor appeared in Nature in 2024 and used feed-forward CNNs, ResNet-style image models, and established PPO experiments.[1] A 2026 PMLR paper supplies the architecture-conditioned reset comparison.[6] An ICML 2026 record for InterpLayers covers continual reinforcement learning and links an author project page, but the conference record does not itself establish a version pin, checkpoint set, or successful reproduction.[10] Paper generation, experimental backbone, and release maturity are separate facts. Any intervention that survives those local limits must re-enter the same two-axis comparison against the eligible arms under the retention and resource constraints.

A promotion rule turns the comparison into a checkpoint policy

Before viewing results, set three thresholds from application tolerance and repeated-run variability: a maximum acceptable new-target deficit, a minimum retained-performance floor, and a maximum resource budget. Do not import a universal numerical threshold from dormancy, rank, or another internal statistic; none is validated across these regimes.

First verify that Arm C can learn the canary within the fixed budget. If it cannot, stop: the canary does not identify checkpoint aging. Then apply the following rule to arms that meet the retention and resource constraints:

1. Continue the aged checkpoint when Arm A's new-target curve is within the predeclared tolerance of the best eligible arm. 2. Reset optimizer and training state, then continue when Arm B materially improves over A and is within tolerance of C without crossing the retention floor. 3. Retrain from scratch when C exceeds both aged-weight arms by the predeclared margin, can be trained on data sufficient to meet the retention floor, and remains inside the resource budget. 4. Promote nothing when no arm satisfies both axes, when apparent differences are smaller than repeated-run variability, or when an uncontrolled change in data, compute, capacity, or tuning prevents a fair comparison.

If C wins the new-target race but cannot retain required behavior because old training signal is unavailable, the comparison has found a diagnostic winner and no deployable winner. The correct outcome is to recover data, relax the retention requirement explicitly, or test a bounded repair—not to relabel the best current-score model as adaptable. This checkpoint policy inherits the regime and parity limits below.

Limitations

The strongest evidence supports loss of later-task learnability in specific long-horizon continual, reinforcement-learning, and stationary warm-start protocols. It does not establish a field-wide prevalence rate, a single mechanism, or one remedy across task families, transition types, architectures, optimizers, replay policies, and training horizons.

More specifically, the studies cited here do not provide a clean, broadly replicated full-parameter experiment demonstrating stable old-target retention together with a fully matched aged-versus-fresh new-target deficit in the same model. The cited intervention studies also do not jointly report fixed-budget new-task learning, retained-task performance, peak performance, compute, memory, and tuning parity.

Protocol parity is a separate limit. The three-arm procedure is a conservative experimental contract assembled from established measurement distinctions and partial controls; complete optimizer-state, update-budget, and tuning-budget parity has not itself been validated as one released protocol.

Dormancy, norm growth, rank, curvature, churn, and spectral measures have useful regime-specific associations, causal ablations, or counterexamples, but no universal threshold or portable predictive lead time. That is the diagnostic boundary. The newest intervention and architecture results include fresh papers and preprints with heterogeneous backbones and incomplete reproduction state. Synthetic gradual transitions establish a controllable experiment, not safety under natural drift.

The open question is whether the same three-arm, two-axis result will replicate across independent production-like data streams while holding retention, optimizer state, compute, capacity, and tuning parity at once.