Training a Language Model to Judge New Questions

Introduction

The first two articles in this Jev series examined the decision-model capability and interface and related methods outside Jev. This third article asks how to train the language-conditioned judgment behind such an interface: how does one model answer questions whose meanings and possible answers arrive with the request?

The essential move is to learn a relationship, not a permanent catalogue of classes. The model must connect the supplied question and option descriptions to the available evidence. Its training must make those words consequential; its probability targets must specify what uncertainty the numbers describe.

Two public resources let us inspect different parts of that process. Laya is the model case: its card exposes how a shared decision head scores the supplied options. [1] Sev is the experiment case: its released repository offers a cross-entropy baseline, adjustable objective weights and training/evaluation commands. [11] We use the visible architecture to understand the computation, and the public baseline to turn the learning question into an experiment.

Three questions organize the path:

How do new questions become model input?

Imagine a ticket classifier whose permanent output slots are billing, technical support and sales. The association between each slot and its class is learned during training. A request-conditioned model instead receives the ticket, “Which team should handle this?”, and descriptions of the available teams. The same ticket can then support a different question: “Has the customer requested a refund?”

Write the desired computation as

where $x$ is the available state, $q$ the question, and $O$ the current options; $\theta$ denotes the scoring parameters shared across requests. The important change is that the trainable computation sees the question and option meanings, rather than receiving only the state and returning a permanent class index.

The encoder turns the words into representations that depend on the surrounding request. The shared scorer then judges each supplied option in that context. In Laya, this relation has a visible implementation: its card describes a fully fine-tuned ModernBERT-large backbone and a decision head trained from scratch; each option is scored at a contextual marker token and the scores are softmax-normalized within that question. [1] Softmax converts the scores into nonnegative values that sum to one across the current options. Thus the same scorer can operate on different option descriptions without adding a class-specific output weight for every new label.

A different implementation can learn the same kind of reusable relation. Yin, Hay and Roth convert candidate labels into textual hypotheses and apply a BERT entailment model to each text–hypothesis pair, with no task-specific labeled examples in their fully unseen-label setting. [2] Here the reusable judgment is whether a text supports a language-defined proposition, rather than which permanent output slot it belongs to.

A new label is now expressible, but a new rule is not automatically understood. “Billing problem” invokes familiar semantics. “Eligible only if the duplicate charge occurred after the revised policy took effect” requires applying a particular condition. The architecture makes those words available to the computation. The next question is how to make them control the answer.

How do we establish that the model uses the question and rules?

During training, the rendered state, instructions and options produce contextual representations. A loss on the requested answer sends gradients back through that computation: the answer error adjusts whichever encoder and scoring-head parameters are trainable. If every example asks the same question, the model can succeed while largely ignoring the instructions. A varied set of judgments over similar states makes that shortcut less useful.

Teach distinctions that the state alone cannot answer

Consider a proposed training family. A message says, “We were charged twice; please return the duplicate payment.” Ask whether a refund was requested, whether one was already received, and which department should handle the ticket. The message stays fixed, but the requested judgment changes.

Now change the message to “We were charged twice, but do not refund it yet.” The requested-refund answer should change; the department may not. Holding the state fixed while changing a policy clause creates another useful pair. These contrasts specify both when an answer should move and when it should remain stable.

As proposed data-design experiments, these pairs aim to make a keyword shortcut fail while a question-sensitive computation succeeds. Include decisive negatives and inapplicable rules, and vary domains, descriptions, lengths and option counts. Audit the rendered tokens too: no training objective can teach the model to use a condition that truncation consistently removes.

Instruction tuning—training on tasks described in natural language—provides evidence that the instructions themselves can matter. In FLAN's ablation, removing instructions or replacing them with dataset names worsened performance on held-out task clusters. [3] Its diversity experiments also found better average transfer as more task clusters were added, though not every added cluster helped. [3]

The relevant diversity is not merely more phrasings of the same familiar decision. The Only-IF studies use controlled rewriting tasks to investigate unseen instruction semantics: diversity restricted to a narrow semantic domain did not guarantee broader generalization. [4] This motivates covering different operations and rules, not just multiplying examples of a few workflows. Neither study establishes that the proposed refund pairs are a sufficient decision-training recipe; they motivate what to vary and what to test.

Test unfamiliar meanings, not just unfamiliar messages

Use a transfer matrix rather than one random split: new cases within familiar workflows, new labels or question families, new rules, and new domains. Build those holdouts before training so that “unseen” refers to the meaning being tested, not simply to a message the model has never encountered.

FLAN provides a precedent for holding out task families and close neighbors, using different held-out-family checkpoints rather than one checkpoint demonstrated on every unseen family simultaneously. [3] For a proposed decision test, fix a state and change the requested rule; fix the rule and change decisive evidence. Evaluate whether the answer follows those changes. Also test when a rule is inapplicable, rather than rewarding a model that always applies the most prominent clause.

Within-workflow adaptation answers a narrower question. Laya's specialist card reports accuracy of 0.766 versus 0.362 for the non-fine-tuned English checkpoint on 400 test cases containing 2,000 decisions, with the specialist trained for four synthetic workflows. [9] These author-reported measurements describe adaptation in those workflows; they do not test unfamiliar-policy transfer.

Presentation tests should distinguish irrelevant variation from a changed problem. Permute nominal options and align their probabilities by meaning; test equivalent descriptions and irrelevant context. Inspect long requests to ensure the decisive words survive. Adding a valid new category can legitimately redistribute mass. Duplicating a meaning requires an explicit allocation convention, and removing the valid answer tests whether the model can reject an incomplete choice set. A trained outside option or separately validated deferral mechanism may help; merely attaching “other” does not establish that it works.

These tests ask whether the model uses the evidence under the requested rule. They still leave the numerical question open: even when it chooses the right answer, what does a reported probability of 0.8 estimate?

What do the reported probabilities mean?

Define the target before collecting labels

Complete the sentence: “This number estimates the probability of ___, given ___.” The event might be an answer under a reference adjudication, a judgment from a specified annotator population, a teacher's reported belief, or a future outcome under a stated resolution rule. These require different data.

An explicit refund request is a judgment about the message; cancellation next month is a forecast about a later event. For human variation, retain repeated judgments and document the population and instructions. For a reference judgment, establish the adjudication rule. For forecasting, record the horizon, resolution criterion and information cutoff, then evaluate against subsequent outcomes. Training on the completed conversation while claiming to forecast an earlier moment would teach hindsight.

Per-case disagreement and confidence across cases are also different quantities. Human label variation asks how people's judgments are distributed for this case; confidence calibration asks whether predictions assigned a given confidence are judged correct at the corresponding rate. [5] Neither interpretation should silently stand in for the other.

Typed-decisions makes this distinction concrete: its dataset card says gold is the mean of three samples from a teacher, and that scores measure agreement with that teacher rather than correctness. [6] Repeated samples from one teacher are not three independent experts. A student matching those distributions learns a judgment surrogate; a field called churn_risk does not turn synthetic teacher judgments into observed customer churn.

Define the possible events as well. Mutually exclusive alternatives can share one unit of categorical probability. Simultaneously applicable properties need separate binary judgments or an explicitly defined joint space; their marginal probabilities need not sum to one. Ordered rubric levels require preserving their order. An expected rubric level is a summary of a distribution, not itself a probability.

Fit the distribution you intend to report

The loss connects that target to the prediction. For target probabilities $t_i$ and predicted probabilities $p_i$, soft cross-entropy is

It penalizes assigning too little probability where the target puts mass. In the unconstrained case, its minimum is the target distribution itself—not merely the target's most likely answer. Thus a soft teacher target trains agreement with the teacher distribution. With one-hot observed labels—targets that put all probability on the single recorded answer—the population objective learns the conditional label distribution rather than copying certainty from each label.

For illustration, a forecast of 0.8 claims an approximately eight-in-ten outcome frequency across comparable cases under the stated information. A teacher assigning 0.8 instead supplies that teacher's reference probability, not an observed outcome frequency. The learner estimates the conditional pattern; it need not assign identical probabilities to unlike cases.

A probability scoring rule is called strictly proper when truthful reporting uniquely optimizes the expected score under the target distribution. [7] That incentive is useful, but its target is still the event or reference process you chose. It cannot transform teacher agreement into correctness, and the ideal optimum does not establish that a finite trained model has reached it.

There is also a distinction between beliefs and actions. Expected-cost decision theory combines class probabilities with the costs of possible actions; choosing the most probable class need not minimize cost. [8] When costs change, change the action rule rather than redefining the reported belief. Softmax alone supplies neither of these semantics: over one option it returns one even when that option is unsupported.

Choose a baseline before adding reinforcement feedback

If a full target distribution is available, direct fitting is the simplest comparison to establish. Distillation describes where supervision comes from—for example, a teacher's probabilities. Cross-entropy describes one way to fit that supervision. Reinforcement learning instead estimates an update from rewards for sampled or perturbed outputs. The question is what that feedback adds to the judgment and distribution you need.

In Laya's described reinforcement update, Gaussian noise perturbs the logits—the option scores before normalization—and rewards for the resulting candidate distributions are compared with the mean reward across that group to estimate an update. [1] Its typed-decisions specialist card explicitly describes adaptation from base Laya using that feedback alongside soft cross-entropy against teacher distributions. [9] This is hybrid training: reinforcement feedback and direct distribution supervision both contribute.

Sev offers a public baseline for that comparison. Its README reports a 30-run comparison of cross-entropy-only (CE-only), reinforcement learning plus cross-entropy (RL+CE), and reinforcement-learning-only (RL-only) configurations, saying CE-only matched or beat the RL-containing alternatives on its proper-score measurements. [11] The practical implication is to include direct distribution fitting, not to declare RL unnecessary for every stage of capability acquisition. The reported comparison needs an exact initialization and update manifest to identify which stage of learning it tests. Without that checkpoint history, it cannot isolate RL's contribution to earlier capability acquisition.

Reward-based learning addresses a different feedback problem when you observe only the consequence of the action you chose, as in the textual decision setting studied by Chen, Zhang and Zhu. [13] Learning from selected consequences requires accounting for which actions were tried; it is not equivalent to matching a precomputed teacher distribution. Optimizing such a policy still does not, by itself, make its belief reports calibrated.

TypeSafe presents Reinforcement Learning for Calibrated Decisions as Jev's calibration-oriented training approach. [14] The public Laya and Sev resources provide inspectable architecture and training cases for investigating that learning goal. Compare additional feedback under a fixed starting point, renderer, data and budget: does it improve the capability and probability target the experiment actually measures?

Run a controlled adaptation experiment

This practical adaptation path uses Sev's released interface as a runnable starting resource. The sequence combines that runbook with proposed experimental controls; it does not assume those controls are already implemented.

1. Record the starting model and update scope. Preserve the actual loaded checkpoint, model/tokenizer revision, known prior training history, renderer and input budgets; label unavailable history unknown. Declare whether the encoder and head are fully updated, only the head changes, or adapters are used. Inspect the configuration and record the implemented loss and its weights. This tells you which learning stage the result can describe.

2. Build request-shaped targets. Preserve state, instructions, described options, question type and the full target distribution in aligned option order. Sev documents a typed-decisions-compatible case with state, questions and gold, flattened into one row per question for training. [11] Decide whether this experiment imitates teacher distributions or estimates another event, and obtain the corresponding labels. Do not replace a full soft target with its hard argmax. The semantic contrasts in the previous section are proposed additions to test.

3. Split before tuning. Keep all questions from a case—and equivalent renderings of the same latent problem—together. Separately hold out task, rule or domain families for the generality claim. Reserve distinct data for model selection, calibration fitting and a final untouched test, and write the checkpoint-selection rule in advance. Sev says its calibration slice comes from train rather than test. [11] Inspect its case grouping and selection/fitting behavior, implementing the required independence wherever needed. Final-test cases and equivalent renderings must not influence any of these choices.

4. Train the target-matching baseline. Work in a checkout of the Sev repository; its quickstart requires Python 3.12 or newer and uv, and publishes a real-data tiny-encoder smoke test and CE-only training overrides. [11]

uv sync
uv run python scripts/training/smoke_test.py
uv run python scripts/training/train.py \
    --config configs/train.yaml --w_rl 0 --w_ce 1

The smoke test checks the pipeline, not broad intelligence. Choose epochs and optimizer settings using the declared training/selection data. Treat RL or another objective as a separately specified comparison: hold initialization, target data, rendering, budgets and evaluation constant, and report variation across runs.

5. Freeze the checkpoint, fit correction, then evaluate. Sev publishes an evaluation entry point that fits temperatures on its calibration slice and reports raw and post-temperature test metrics. [11]

uv run python scripts/training/evaluate.py \
    --ckpt checkpoints/<run>/best.pt --json results/eval.json

Select the checkpoint before inspecting the final test. Verify the evaluator against the split rules above, and preserve both raw and corrected probabilities.

Evaluate the numbers against their stated meaning

Against observed outcomes or reference adjudications, report log loss, Brier score and reliability diagnostics, including important subgroups. Log loss penalizes low probability on the realized answer; Brier score measures squared probability error. Reliability diagnostics compare reported confidence with observed correctness rates. Against soft teacher or human distributions, measure distributional error against that reference and state whose judgments it represents.

Include a state-blind prior: typed-decisions itself notes that its prior can have low expected calibration error (ECE) while ignoring the input. [6] ECE summarizes the gap between reported confidence and observed correctness rates. A model can look calibrated by this measure without using the request, which is why numerical evaluation must accompany the semantic tests above.

Temperature scaling divides logits by a fitted positive scalar; Guo and colleagues optimize it on validation negative log-likelihood, and it preserves the winning class. [15] It can correct confidence but cannot repair an incorrect ranking. Ovadia and colleagues demonstrate that calibration on an in-distribution validation set does not guarantee calibration under dataset shift in their image, text and categorical-classification studies. [16] These adjacent-regime findings motivate checking probability quality again on the unfamiliar-rule and domain holdouts, rather than carrying a familiar-workflow calibration result across them.

If the application escalates uncertain cases, evaluate error versus coverage on held-out cases under the intended deployment conditions. Do not substitute distribution concentration for a correctness estimate: Laya's helper defines its confidence_from_probs as normalized entropy confidence, $1-H(p)/\log k$, not a demonstrated probability that its answer is correct. [10] Here $H(p)$ is the distribution's entropy and $k$ is the number of options. A confident-looking distribution is not a validated automation threshold.

The experiment should therefore end with two linked results: how judgments respond to unfamiliar meanings, and how the reported probabilities perform against their declared target on those same conditions.

Conclusion

To serve as a reusable decision model, a language model must learn to connect evidence with the currently requested judgment and language-defined answers. Shared scoring makes new choices representable; question-sensitive supervision and semantic-transfer tests establish whether their meanings are actually used.

Meaningful probability adds a separate demand. Define the event or reference process, fit its distribution, and evaluate the resulting numbers independently of answer accuracy. Laya makes a scoring architecture visible, and Sev supplies a practical baseline for controlled adaptation. Their value is to make the training questions inspectable—not to settle generality by schema design or calibration by an algorithm's name.

Limitations

This account draws on released model cards, a reward helper and runbooks. It does not independently reproduce the reported Laya/Sev measurements or verify Laya's detailed notebook training cells and Sev's linked loss/report methods, per-run results, initialization or split implementation. Jev's public launch description leaves its detailed targets and updates undisclosed.

Typed-decisions teaches and evaluates teacher agreement; the exact teacher endpoint and generation implementation remain unresolved. Instruction, rule-diversity and shifted-calibration studies come from different model/task regimes. Their mechanisms inform experimental design, while transfer to the intended decision setting requires its own measurements.

Establishing broad capability acquisition and calibrated behavior on new policies requires target-specific implementation inspection and independent transfer/outcome evaluation. Finite data, model capacity, optimization and population change constrain the ideal probability objectives. Update scope, data mixture, annotation counts, hyperparameters and the contribution of RL remain empirical questions.