From Language to Incremental Edge: Engineering LLM Semantics into Quantitative Models

In The LLM Edge in Finance Is a Division of Labor, we argued that an LLM’s most defensible edge is not to replace numerical forecasting, portfolio optimization, risk, or execution, but to turn the meaning in filings, earnings calls, news, and research into information the quantitative stack can test.

This report takes the next step: how should those semantic measurements be constructed, combined with strong point-in-time numerical models, and evaluated to determine whether they add genuine incremental edge?

What would count as incremental edge?

A model can read an earnings call correctly and still contribute nothing to an investment process.

It may notice that management lowered guidance, changed the basis of comparison, avoided a question about demand, or described a constraint that has not yet appeared in reported numbers. Those observations can be economically meaningful. But unless they change a forecast beyond what prices, fundamentals, consensus estimates, and market conditions already imply, they are interpretations—not edge.

But “let the LLM handle language” is only an architectural slogan. A real system still has to decide what the language model should measure, how that measurement becomes a feature or forecast, where it meets the numerical model, how uncertainty is calibrated, and what happens when an equally well-tuned numerical-only system performs just as well.

The useful unit of design is therefore not the LLM agent. It is the complete path from a document available at time $t$, through a reproducible semantic measurement, to a forecast and decision that can be evaluated after $t$. Once that path is explicit, feature fusion, residual models, forecast ensembles, and portfolio views become testable alternatives rather than fashionable labels. More importantly, the resulting experiment can answer the only question that matters here: did language provide new information, or merely redescribe information the quantitative system already knew?

The central estimand is one conditional prediction. It cannot by itself settle document availability, model-training leakage, execution, capacity, or prospective replication; those remain separate conditions developed below.

At decision time $t$, let $X_t$ contain everything in the numerical information set: prices, volume, point-in-time fundamentals, consensus expectations, factor exposures, liquidity, and market state. Let $D_t$ contain only documents that were actually public and processable by that time: news, earnings calls, regulatory filings, and research text. An LLM or another language model transforms those documents into a semantic measurement,

which might be a structured event, a vector of features, an embedding, a portfolio view, or a forecast. A numerical-only model estimates $f_0(X_t)$; the augmented model estimates $f_1(X_t,Z_t)$. The object worth calling “LLM edge” is

This definition makes the burden of proof explicit. Semantic competence is neither a forecast nor a trade. Text adds value only if it changes the conditional distribution of a future target after the conventional information set is known. The estimate must also begin at a feasible decision clock and remain valuable after risk, turnover, market impact, borrow, capacity, data, and inference costs.

That leads to a modular architecture:

The LLM is best viewed initially as a measurement instrument for language. It can be promoted to a larger forecasting role if the evidence supports that choice, but arithmetic, calibration, risk estimation, and portfolio construction should remain explicit numerical operations. This is a testable division of labor, not a claim that every modular hybrid will beat every end-to-end model.

Start with the target and the clock

Before choosing an embedding or a fusion layer, specify exactly what is predicted and when the prediction could have been acted upon. For an equity event strategy, a useful target is

where $t$ is the system's document-availability time, $\ell$ includes ingestion, inference, order-routing, and market-opening delay, and $F$ removes market, industry, style, or other chosen risk exposures. This makes $Y$ a residual return over a tradable window rather than a raw move that may already have occurred.

Three targets must not be conflated:

• Immediate price discovery: the return from release until the first executable quote. This can establish that the model measures how markets interpret an event, but normally cannot establish a strategy's attainable alpha.
• Post-event drift: the return after the first feasible trade. This is the appropriate target for a short-horizon trading claim.
• Fundamental outcomes: later earnings surprises, margins, sales, volatility, default, or analyst revisions. These can be better tests of whether an extracted event captures the proposed economic mechanism.

This distinction is not pedantic. A real-time agentic-AI benchmark around earnings announcements evaluates contemporaneous return explanation using information available at announcement time. That is valuable evidence about semantic measurement and price discovery, but its target does not by itself establish post-execution alpha.[11] Similarly, if a headline says that shares fell after guidance was cut and the label includes that already-realized fall, the model is reading the answer from the document rather than forecasting the future.

The target specification should therefore include the asset universe, event-time convention, forecast horizon, execution lag, return adjustment, label-maturity rule, and cost model before model selection begins. For overlapping multi-day targets, training and evaluation windows must be separated so that no training label reaches into the test period. The same document, horizon, and trade clock must be used by every baseline.

Build the numerical model that semantics must beat

A credible benchmark estimates

where $M_t$ is the common market state. The contents of $X$ depend on the target, but a serious equity benchmark commonly includes:

• price and volume features such as momentum, reversal, gaps, volatility, volume surprise, liquidity, spreads, and order-flow proxies;
• point-in-time fundamentals and expectations such as valuation, profitability, investment, accruals, leverage, operating trends, analyst consensus, and realized-versus-expected surprises;
• market and regime variables such as rates, credit spreads, volatility, market liquidity, sector conditions, and macro releases;
• cross-sectional relationships such as size, industry, factor exposure, supply-chain links, and each signal's rank within the contemporaneous universe.

The benchmark should not be artificially linear. Large-scale empirical asset-pricing comparisons find that trees and neural networks can exploit nonlinear interactions and repeatedly identify momentum, liquidity, and volatility as important predictors.[1] The practical implication is that an LLM feature must beat a well-tuned nonlinear numerical model, not just the historical mean or a small regression.

Alpha and risk models should remain separate. Text may improve expected returns or event-risk forecasts; covariance, factor exposure, liquidity, and portfolio constraints are usually better estimated from numerical histories. Holding the risk model fixed while comparing alpha models also makes attribution cleaner.

Turn documents into an auditable semantic data product

The economically meaningful object is not the prose itself but a latent event: a guidance revision, a supply interruption, a regulatory action, a change in pricing power, a management-credibility signal, or a cross-firm propagation channel. The language model supplies a noisy measurement of that event:

A reliable implementation separates provenance, extraction, deterministic transformation, and economic prediction.

Preserve document provenance

For each source artifact, retain event time, publisher time, vendor receipt time, system receipt time, version or correction status, document identifier, issuer/entity mapping, source, language, and document type. Also archive the exact model version, prompt or extraction specification, retrieval snapshot, input hash, and output hash. Story-chain and duplicate identifiers matter: ten syndicated copies of one article are not ten independent signals.

Entity resolution must itself be point-in-time. A historical article can be linked only to the ticker, issuer, industry, and supply-chain relationships known then. Corrections and withdrawals should create new versions rather than silently overwriting the text used by an earlier forecast.

Extract constrained events, not investment essays

A useful first representation is a fixed schema whose fields can be scored independently, for example:

{
  "entity": "issuer_or_related_asset",
  "event_type": "guidance_revision",
  "metric": "operating_margin",
  "direction": "down",
  "magnitude_text": "reported_range",
  "comparison_basis": "prior_company_guidance",
  "horizon": "next_fiscal_year",
  "certainty": "explicit",
  "novelty": "estimated_score",
  "relevance": "estimated_score",
  "channels": ["cash_flow", "supplier_demand"],
  "evidence_spans": ["supporting source span"]
}

Document-level financial event extraction predates modern general-purpose LLMs: systems such as Doc2EDAG represent entities and event roles across sentences and score structured event records directly.[2] The lesson is architectural, not historical: auditability comes from predefined roles, entity linkage, and source-aligned spans, not from a fluent free-form rationale.

Arithmetic should be executable. The language model can locate quantities, identify their units and comparison basis, and propose an operation; deterministic code should compute the surprise, percentage change, aggregation, or valuation feature. TAT-QA formalizes this separation by selecting relevant table cells and text spans and then applying symbolic aggregation operators.[3] More recent finance evidence also documents sharp deterioration when LLMs must tally many values or reason over full statements with irrelevant information.[4] The safe interface is therefore source spans + typed quantities + program, followed by deterministic execution and validation.

Use two kinds of supervision

The measurement model and the economic model answer different questions and need different labels.

1. Semantic labels test whether the event was understood: entity and event precision/recall, direction, magnitude bucket, horizon, comparison basis, causal role, and whether the cited span supports the output.
2. Economic labels test whether a correctly measured event predicts returns, earnings, volatility, or another future outcome after conditioning on $X$.

This prevents a common category error: better sentiment classification or event extraction does not prove investment value. It also makes failure diagnosable. A feature can fail because the event was extracted incorrectly, because the event was already priced, or because the downstream estimator was poorly specified.

Convert events into stable numeric features

From the structured record, construct low-dimensional features such as event probabilities, signed direction times relevance, surprise relative to prior guidance or contemporaneous consensus, novelty relative to the issuer's prior documents, certainty or hedging, management–analyst disagreement, Q&A responsiveness, propagation exposure, and horizon-specific event intensity. Dense embeddings can be retained as an alternative representation, but they should not replace the auditable features by default.

For multiple documents, aggregate only with weights learned from earlier data:

The essential point is that source reliability, novelty, and decay are explicit and estimated out of sample. Different event types deserve different half-lives. A one-day analyst action and a multi-quarter regulatory constraint should not share a mechanically chosen decay.

Choose fusion by what must interact

Keep one machine in view first: a cross-fitted numerical baseline, a matched residual learner, and the semantic system running as a challenger. The earlier guidance-cut case makes the sequence concrete. Once the fixed schema records direction, comparison basis, horizon, and evidence spans, the residual test asks whether that measurement forecasts what the numerical baseline missed.

The other fusion designs are conditional satellites, introduced only when the interaction under test requires them. Feature-, representation-, forecast-, and view-level describe where information meets. Residual describes the prediction target. Challenger describes an operational role. Ensemble describes a combination rule. They can coexist: a residual model can use feature-level inputs and later become one member of a forecast ensemble.

There is no evidence-based universal ranking. One chronological financial study reports that a specialized architecture with modality-specific processing beats several unimodal and multimodal alternatives, while also finding that where cross-modal attention is introduced matters.[5] The same evidence base contains cases in which naïve multimodal inputs do not beat text alone. The conclusion is that interaction structure must be tested under matched data and clocks—not that “more modalities” is intrinsically better.

View-level fusion is mathematically clean when the inputs are genuine views. In a Black–Litterman-style update,

Here $P$ identifies assets or relative positions, $q$ contains expected returns, and $\Omega$ represents view-error uncertainty; smaller view variance gives the view more influence.[6] For an LLM, $P$, $q$, the horizon, and $\Omega$ must be learned and validated from historical out-of-sample errors by event type, source, and regime. A model's statement of high confidence is not a calibrated return distribution.

Why residual learning is the cleanest first experiment

Residual learning directly targets the conditional increment. First generate walk-forward or cross-fitted baseline forecasts, never fitted values from the same observations:

Then fit two residual learners under identical cross-fitting, capacity, tuning budget, timestamps, and costs:

The numerical residual learner $h^{(X)}$ is the control for the semantic residual learner $h^{(X,Z)}$. Attribute semantic lift only to the incremental performance of $h^{(X,Z)}$ over the same-capacity $h^{(X)}$, not to the augmented system's gain over the baseline alone.

An even stricter diagnostic residualizes the semantic features against the numerical state as well:

If $\widetilde Z$ cannot forecast $\widetilde Y$, the apparent semantic signal may only be a repackaged momentum, size, volatility, sector, or attention feature. Residual learning is a strong starting test because it asks the right incremental question; the current comparative literature does not establish it as the universally best production architecture. The guidance-cut case now has a falsifiable challenger; before interpreting its performance, validate the measurement, calibrate the forecast, and evaluate the decision separately.

Validate measurements, calibrate forecasts, and evaluate decisions

The three stages need separate checks: measurement validation assesses semantic accuracy, forecast calibration remaps predictive outputs, and decision evaluation measures utility under a fixed rule.

Measurement validation

On an independently annotated, time-stratified sample, report event and entity precision/recall, direction and magnitude accuracy, horizon accuracy, and evidence-span support. Break the errors down by year, industry, language, document type, source, event novelty, and document length. A generated explanation with no source-aligned evidence should not become a production feature.

Forecast calibration

Map scores to probabilities, means, quantiles, or full predictive distributions using a rolling validation set that precedes the test period. Platt scaling, isotonic regression, temperature scaling, or hierarchical shrinkage may be appropriate depending on sample size and output type; the calibrator is itself a fitted model and must obey the same time boundary.

Use Brier score or log loss for binary events, quantile loss for quantiles, and CRPS or another strictly proper score for distributions. Strictly proper scoring rules are designed to reward honest probability distributions rather than strategically exaggerated confidence.[7] Report reliability and sharpness together, plus calibration slope and interval coverage by event type and regime. Aggregate calibration can hide a model that is useful for large caps and inverted for small caps.

Before paying to acquire an expensive semantic feature at scale, run a viability test: fit the frozen downstream mapping on the validation data, perturb the proposed feature over prespecified values, and verify that the resulting forecast changes materially. If the downstream predictor, stacker, or gate assigns the LLM no effective weight, it has rejected the feature; more scoring volume cannot rescue a signal the frozen forecast ignores.

Decision evaluation

Finally evaluate the forecast through a fixed decision rule, for example

subject to beta, sector, factor, liquidity, position, and short-availability constraints. Forecast accuracy can improve while decision utility falls because the improvement occurs in untradeable names, arrives too late, or demands excessive turnover. Trading-cost evidence from live institutional orders shows that implementability and capacity depend on liquidity, trade size, and portfolio design rather than a universal cost haircut.[16] The economic endpoint should therefore be incremental certainty-equivalent return or another predeclared net utility measure, not gross spread alone.

Enforce two point-in-time clocks

These checks are meaningful only if both the inputs and the model obey the clocks that generated them. Data point-in-time and model point-in-time are independent requirements.

Data point-in-time

Every observation needs both the period it describes and the time it became available. A fiscal-period end is not the availability time of a filing. The SEC's EDGAR APIs update submissions and XBRL data as filings are disseminated and explicitly document different processing delays for the two feeds.[8] Macro series must use release vintages: ALFRED archives the real-time periods in which observations were released and later revised.[9] Today's final macro value, consensus reconstruction, or corrected filing cannot be inserted into an earlier forecast origin.

The same logic applies to the historical universe, delisted securities, ticker mappings, analyst estimates, document corrections, news syndication, transcript release, time zones, and observed vendor-to-system latency. Store the as-of snapshot used by each forecast rather than assuming it can be perfectly reconstructed later.

Model point-in-time

A data-clean backtest can still be contaminated by the language model itself. A model analyzing a 2015 headline in 2026 may have absorbed later outcomes during pretraining or post-training. A provider's nominal knowledge cutoff attests, at best, one part of the training pipeline; finance experiments find that look-ahead artifacts can materially inflate apparent performance and that numerical-reasoning weaknesses can coexist with high task scores.[4]

Company anonymization is a useful negative control, not a certificate of cleanliness. Glasserman and Lin compare named and anonymized financial headlines and find, within the model's training window, that anonymized headlines can perform better because company knowledge distracts from text sentiment; the experiment illustrates that memorization and distraction are separate effects.[10] An embedding is no safer merely because no return question was asked: future knowledge can be encoded in the representation.

The hierarchy of evidence is therefore:

1. a time-indexed model whose pretraining and post-training cutoffs are auditable;
2. a frozen released model evaluated only on events occurring after the complete model release, with retrieval, prompt, calibrator, and rule also frozen;
3. consistently anonymized inputs used identically in research and live operation, as a diagnostic mitigation;
4. a contemporary proprietary model replayed on events inside its training window, which is exploratory evidence only.

The strongest feasible test is a precommitted forward shadow: freeze model, prompt, retrieval corpus, schema, calibrator, baseline, portfolio rule, and code hashes; timestamp signals before outcomes; and prohibit tuning on shadow results. Historical robustness can reject known failure explanations. It cannot prove permanence, especially in reflexive markets where adoption changes prices.[11]

Run one matched walk-forward experiment

With both clocks enforced, a credible experiment is a sequence of untouched future blocks, not random row cross-validation:

If targets overlap, purge observations whose labels cross the boundary and use an embargo appropriate to the horizon. Preserve dependence when estimating uncertainty: same-day stocks, repeated issuer events, and syndicated stories are not independent rows. Date blocks, issuer/event clusters, or a block bootstrap are more credible than headline-level IID standard errors.

Use a baseline ladder, not one straw man

Run all competitors on the same universe, timestamps, target, compute or tuning budget, and cost model:

1. historical mean or no-trade;
2. regularized and nonlinear numerical baselines;
3. dictionary sentiment and TF–IDF/bag-of-words with a regularized head;
4. a supervised return-specific text model;
5. a smaller domain encoder such as FinBERT-family models;
6. LLM event features, embeddings, and direct forecasts separately;
7. the numerical model plus each semantic representation;
8. any residual, forecast-pooling, view, gated, or end-to-end system.

A smaller financial encoder is not a ceremonial baseline. Recent classification evidence reports that a fine-tuned bidirectional finance encoder can outperform much larger decoder LLMs on discriminative financial tasks, although that result does not establish return prediction.[12] The point is to test whether the proposed edge needs compositional semantics and scale, or merely better domain vocabulary and supervision.

Use ablations that can falsify the semantic story

For a residual design, the core comparison is the matched $h^{(X)}$ versus $h^{(X,Z)}$ pair under identical cross-fitting, capacity, tuning budget, timestamps, and costs. For other architectures, compare $X$ with the identical model augmented by $Z$. Add controls that target specific alternative explanations:

The ablation should match the claimed mechanism. If the claim is understanding negation, novelty, multi-entity causality, or Q&A disagreement, the gain should concentrate in examples that require those capabilities and disappear under sentence shuffling or neutral boilerplate. Once those controls expose what the gain depends on, the next question is whether it survives statistical, regime, and economic tests.

Demand statistical, regime, and economic evidence together

Forecast improvement can be summarized by incremental out-of-sample fit,

alongside loss differences, rank IC and decay, calibration metrics, and results by liquidity, capitalization, event type, and horizon. Report confidence intervals around effects, not only p-values.

The inference method must match the comparison. A numerical model and the same model augmented with semantics are nested, so ordinary non-nested forecast tests are not automatically valid. Forecast-method research documents finite-sample distortions for nested, multi-step comparisons under serial correlation and proposes block/residual bootstrap procedures for those settings.[13] When many candidate models all nest one benchmark, reality-check-style nested comparison methods are available.[14] For genuinely non-nested forecasts, use a dependence-robust loss-differential test; with many models, use a multi-model procedure rather than a collection of uncorrected pairwise tests.

Multiplicity includes every prompt, model version, horizon, universe, event filter, aggregation window, fusion weight, portfolio cutoff, cost assumption, and stopping rule tried. Keep a permanent trial ledger, predeclare a primary specification, and apply family-wise, false-discovery, or superior-predictive-ability correction as appropriate. Sharpe estimates should also be adjusted for selection across trials and non-normal returns; the Deflated Sharpe Ratio was designed for precisely those two inflation channels.[15]

Regime evidence should ask whether the aggregate result is driven by one crisis, a few mega-cap earnings events, one vendor, or the best month. Report rolling performance and leave-cluster-out results across volatility, rates, liquidity, capitalization, sector, source, language, and event type. Remove the best-performing periods and event clusters. Add contemporaneous returns, volume shocks, analyst surprises, and numerical regime variables to test whether the semantic score remains conditionally informative.

Finally, trace the economic mechanism. If complex negative news is hypothesized to diffuse slowly, the effect should be stronger where attention or arbitrage capacity is limited and should decay after release. A pre-release signal suggests leakage or endogenous document timing; an effect confined to the release-to-first-quote interval suggests price-discovery measurement rather than attainable drift.

Promotion should require all of the following:

• both data-PIT and model-PIT integrity;
• positive, interval-supported out-of-sample improvement over strong numerical, classical-text, and smaller-encoder baselines;
• semantic measurement accuracy and source spans consistent with the proposed mechanism;
• survival after multiplicity correction, negative controls, and leave-cluster-out tests;
• positive net utility under conservative latency, liquidity, impact, borrow, capacity, data, and inference costs;
• reproduction in a frozen forward shadow with predeclared pass/fail bounds;
• a stopping rule that reduces weight or reverts to the champion when calibration, IC, net utility, or data quality breaches a threshold.

Synthesis

Semantic capability becomes a quantitative signal through a chain of separately testable transformations:

The most defensible first implementation is therefore modest. Use the language model to extract a fixed, evidence-linked event schema; use deterministic code for magnitudes and surprises; attach the resulting features to a strong walk-forward numerical baseline, preferably beginning with a cross-fitted residual experiment; calibrate the forecast on an earlier rolling window; and run it as a frozen challenger before it can influence capital.

That sequence does not assume the modular design will win. It creates the cleanest opportunity for it to fail. If structured events add nothing beyond numerical and classical-text controls, stop. If only a dense embedding works, raise the evidence bar for memorization and mechanism. If the downstream mapper gives the feature zero weight, do not acquire more of it. If gross accuracy does not survive the execution clock and costs, call it semantic measurement rather than alpha. Only a signal that survives these tests has earned the narrower claim of incremental edge under the evaluated conditions.

Limitations

No historical study can prove that an LLM signal is permanent. Markets are adaptive, document feeds and model versions change, and a forward shadow establishes performance only over its observed period. The available evidence also does not provide a single point-in-time financial experiment comparing every fusion family—feature, representation, forecast, view, residual, challenger, ensemble, and end-to-end—under one target and cost model. Residual learning and challenger deployment are therefore recommended as clean tests and controls, not as empirically established universal winners.

Evidence is uneven across components. Official filing and macro sources support availability and revision clocks, while equivalent provenance standards for every news, transcript, and research vendor remain implementation-specific. Structured extraction and executable arithmetic are well motivated, but much of the directly auditable event-extraction literature predates current general-purpose LLMs. Some recent multimodal and finance-encoder results are preprints or concern classification rather than returns. Finally, transaction-cost studies establish that costs and capacity are portfolio-dependent; they do not supply a universal cost model for an LLM semantic strategy. Those gaps strengthen the case for matched local experiments and prospective evaluation rather than weakening the proof standard.

Passing these model-integration and evaluation tests does not confer deployment approval, operational risk-control ownership, execution permission, or authority to intervene in production. Those questions belong in a companion governance specification; forecast evidence alone cannot assign that authority. The open question is prospective: when the cross-fitted residual system is frozen and run forward as a challenger, does its semantic increment deliver positive net utility under the predeclared clock, costs, and stopping rule?