What Stops Working Before the Million-Token Window Runs Out

What the advertised figure actually denotes

Before any comparison of advertised versus effective, it matters what each vendor's number is a claim about, because the four figures at the top of the market denote four different things.

• OpenAI, GPT-4.1 (April 2025): 1M is the API input limit and also a training claim — the launch page states the model was "trained to reliably attend to information across the full 1 million context length" [14]. The successor GPT-5 line (August 2025) served a maximum of 272,000 input tokens and 128,000 output tokens, a 400K total — smaller than its predecessor's window [15].
• Google, Gemini 2.5 (2025): 1M is the standard API serving limit, with the vendor's own long-context documentation conceding that beyond single-needle lookup "the model does not perform with the same accuracy" and that "performance can vary to a wide degree" [17].
• Anthropic, Claude Sonnet 4 through Opus 4.6 (August 2025 → March 2026): 1M began as a beta tier and became the default at standard pricing for Claude 4.6-generation models — the pricing documentation states that a 900K-token request is billed at the same per-token rate as a 9K-token request [20].
• Meta, Llama 4 Scout (April 2025): the advertised "industry-leading context window of 10M" is a length-generalization extrapolation by the vendor's own account — the same announcement states Scout was pre-trained and post-trained at 256K [18].

Two further complications keep the advertised figure from serving as a ruler. First, "effective context length" has no agreed definition: one benchmark defines it against a fixed absolute baseline, another against 85% of each model's own short-context score — the same model earns a different effective length under each [2, 3]. The Context Arena leaderboard sidesteps length altogether, compressing performance across length bins into a single area-under-curve score weighted by bin width — a number that yields no length at all [28]. Second, the figure became a priced product tier in 2026: a March 2026 practitioner survey of the vendor documentation records GPT-5.4 restoring 1M as a premium tier at double billing with 272K as standard mode [21], while Anthropic went the opposite way: its current pricing page includes the full 1M window at standard pricing — a 900k-token request bills at the same per-token rate as a 9k-token one [20]. When one vendor charges double for the same tokens past a threshold and another does not, the spec-sheet number is a business decision, not a measurement.

Retrieval is not one skill: where each task family stops working

The field's own taxonomy distinguishes retrieval (literal versus semantic), aggregation, and multi-step reasoning as separate capabilities, and a 59-model study found the categories correlate weakly with one another — no single number summarizes long-context ability [6, 4]. The families fail in a consistent order as the window fills, and the order is the useful fact.

Except for NoLiMa's thresholded effective lengths, the entries in this table are task scores at stated lengths. They demonstrate model-specific differences at those lengths; they do not identify how much of any window is usable, which no score at a single length can.

The one gap that genuinely closed

Single-needle retrieval is the success story, and it is recent. In April 2024, RULER found that almost all models fell below threshold before their claimed lengths — despite near-perfect scores on the vanilla needle test, only half of seventeen models held up at a claimed 32K [2]. By May 2026, a small exploratory pilot on a million tokens of classical-Chinese haystack — a single corpus, 18 single-needle cells per model with one query each, using altered needles that contradict training priors to rule out answering from memory, and described by its authors as an exploratory pilot rather than a benchmark — recorded Gemini 3.1 Pro, Claude Opus 4.7, and GPT-5.5 at 18/18 [10]. Anyone whose mental model of long context was formed by 2024-era failures should update in this direction, with the sample size in view: for finding one literally-stated fact, the advertised window of these three models held at every depth the pilot tested.

That is also why the needle test stopped being informative. Vendor documentation advertises ~99% single-needle recall at 1M [17], and OpenAI's GPT-4.1 launch reported perfect needle retrieval "at all positions and all context lengths, all the way up to 1 million tokens" [14]. A saturated test distinguishes nothing; the load-bearing question is what happens on the tasks buyers actually run.

The first thing to fail — and the stalest measurement

When the question and the passage that answers it share no convenient vocabulary — when retrieval requires inferring that "the character who ate sushi" is the one described as dining in Tokyo — performance collapses at lengths that are small fractions of any advertised window. NoLiMa, whose original February 2025 cohort was thirteen models claiming at least 128K, found effective lengths (the longest context at which a model holds 85% of its own short-context baseline) generally at or below 2K tokens in that cohort; eleven of thirteen models fell to half their baseline by 32K [3]. GPT-4o, the strongest exception, still failed to generalize beyond 8K, falling from a 99.3% baseline to 69.7% at 32K [3]. Gemini 1.5 Pro, advertising 2M tokens, measured an effective length of 2K — a thousandfold gap [3]. Extended runs at 128K left GPT-4o above half its baseline while Gemini 2.0 Flash, advertising 1M, dropped to 16.4% [3]. The study's extended results, updated through July 2025 under the same 85% criterion, add one newer model: GPT-4.1, advertising 1M, measures 16K effective — an order of magnitude above the original cohort's typical figure, and still some sixty-fold short of its claim [3].

Reasoning models do not escape this. On NoLiMa's hardest questions, o1 fell from 99.9 to 31.1 and o3-mini to 18.9 as context grew to 32K (early-2025 measurements on the then-current reasoning generation); the authors conclude that models "enhanced with reasoning capabilities or CoT prompting struggle to maintain performance in long contexts" [3].

The uncomfortable epistemic fact: the task family that fails first has the stalest numbers. The thresholded record for this family currently ends at GPT-4.1 (April 2025); no NoLiMa-class measurement on the GPT-5.5/Opus 4.7/Gemini 3.1 generation — the one whose literal retrieval looks strongest — appears in the cited literature. Whether non-literal retrieval improved alongside literal retrieval on that cohort is, as of mid-2026, an open question, not a settled one.

The vendors' own tables show the cliff

The strongest evidence that the gap survives at the top of the market comes from the vendors themselves. OpenAI's GPT-4.1 launch page pairs its perfect needle chart with its own harder instruments, and publishes both ends of each curve: two-needle MRCR at 57.2% at 128K falling to 46.3% at 1M, and Graphwalks — the multi-hop benchmark OpenAI describes as one that "cannot be solved sequentially" — at 61.7% within 128K falling to 19.0% beyond it [14]. The GPT-5 launch data (August 2025) shows two-needle MRCR falling from 95.2% at 128K to 86.8% at 256K — measurable degradation within a third of the previous generation's advertised window, on the vendor's own benchmark [15]. Google's Gemini 2.5 technical report (July 2025) shows 8-needle MRCR-V2 for Gemini 2.5 Pro at 58.0% at ≤128K falling to 16.4% at 1M, and its hard retrieval suite falling from 87.0% to 69.8% over the same span [16].

At the March 2026 frontier — the first moment all three vendors offered 1M outside a beta tier, Anthropic's 1M having been beta-only until its March 13 general availability — a practitioner survey compiled vendor system cards, OpenAI's published figures, and the independent Context Arena leaderboard. It put 8-needle MRCR v2 at 1M at 76.0% for Claude Opus 4.6, 36.6% for GPT-5.4 (512K–1M band), and 24.5% for Gemini 3 Pro. These figures are secondhand compilations of primary sources, and the vendors ran the benchmark at different reasoning-effort settings, so the triplet is not a same-conditions ranking — but the spread itself is the finding: on the hardest common instrument, scores at 1M range from roughly 25% to 76% depending on which model, a threefold spread invisible on the spec sheet [21]. These are task scores at one length, not usable fractions of the window — turning them into an effective length would require an explicit threshold and a score-versus-length curve, which this compilation does not supply.

Multi-hop chaining shows the same model-dependence with sharper geometry. The same May 2026 pilot's three-hop test found three decay signatures among its five 1M models: a stable regime (Claude Opus 4.7, and Gemini 3.1 Pro with an unbroken 5/5 at each of 256K, 512K, and 1M), a late cliff (GPT-5.5 and Qwen3.6-plus dropping sharply between 512K and 1M), and smooth decline (DeepSeek V4 Pro). Its conclusion is the one to carry: "nominal context-window length is a poor proxy for usable long-context multi-hop capability," with the 512K→1M transition the sharpest discriminator among the models it tested. The test is statistically thin — five three-hop chains per length tier, one query each — so its per-model figures deserve less weight than its shape [10].

The 2023–24 canon, re-tested

Context-engineering folk wisdom was built on a handful of 2023–24 findings, and each has since been re-tested on later generations. Three canonical claims, three different fates.

Lost in the middle: an effect that fades as the window fills

The July 2023 finding — a U-shaped position curve in which mid-context placement of relevant information tanked accuracy on GPT-3.5-Turbo, Claude 1.3, and contemporaries [1] — does not survive in its textbook form. An August 2025 re-measurement across models with context windows from 8K to 128K found the effect is strongest when the input occupies no more than half the model's window and "declines sharply" beyond that, with primacy bias giving way to recency bias as inputs approach the limit; this reconciles why 100K-token benchmarks kept failing to detect a curve that 2023-era protocols reliably produced [8]. A 2026 reproduction in a retrieval-augmented setting on LLaMA-3.x models did not recover a U-shape at all: "accuracy remains comparatively flat across positions, with only a slight upward trend" [9].

Verdict: weakened — real but conditional on relative input length, and model-specific. One critical boundary: no re-run of the original position-sweep protocol at or near 1M on a 2025–26 frontier model appears in the cited literature. "Models are lost in the middle" stated undated, as a current property of frontier models at frontier lengths, is unsupported; so is its confident negation.

The claimed-vs-effective gap: closed for literal lookup, larger everywhere else

RULER's April 2024 finding that models break before their claimed lengths [2] survived re-testing, with its form transformed: the literal-retrieval half of the gap closed (above), while the non-literal half proved far larger than RULER-era numbers suggested — effective lengths of 2K against claims of 2M, not "about half" [3]. A 2024 analysis of why open-source effective lengths "typically not exceeding half of their training lengths" attributes the shortfall to the left-skewed distribution of relative positions seen in training [7] — a mechanism-level reason to expect the gap to persist by default rather than dissolve with scale.

The degradation-with-length curve itself also survived onto current models: Chroma's July 2025 report ran eighteen models including the Claude 4, GPT-4.1, Gemini 2.5, and Qwen3 generations on deliberately minimal tasks and found performance degrading with input length across all of them, with a single topically-related distractor measurably hurting and four compounding the damage. Its oddest result doubles as a mechanism probe: across all eighteen models, performance was better on shuffled haystacks than on coherently structured ones — coherent surrounding text makes an inserted fact harder, not easier, to isolate [13].

A caveat about the measurements themselves

An August 2026 methods paper complicates part of this literature in the other direction. Testing Claude Haiku 4.5, Sonnet 4.6, Opus 4.7, and GPT-5.5 under two truncation protocols, it showed that the common "shorter context is better" experimental design — truncating from the middle of the prompt — confounds removing distractors with destroying the signal: at 25% retention under naive middle-truncation, the answer-bearing fact sentences survived in fewer than 1% of samples on its BABILong benchmark, and the gold subgraph in 26% of samples on its GraphWalks graph-search benchmark — scores collapsed on both for that reason — whereas distractor-aware truncation at the same retention preserved or improved performance [11]. The degradation findings above rest on designs that vary length while preserving the needle, so they stand — but any specific "longer context hurt us" number that does not state how it separated signal from distractor is ambiguous between two opposite hypotheses.

The revealed effective window: what buyers actually budget

If the measured gap were academic, the systems built on these models would run to the advertised limit. None of the ones with published defaults do.

A user-filed GitHub issue against Claude Code reports automatic compaction triggering well before the window's end even with the setting disabled — a single issue report, closed without action, so evidence about one user's sessions rather than a documented product spec [23]. The pi coding agent defaults its direct OpenAI routes to a 272,000-token window explicitly so requests stay inside OpenAI's short-context pricing tier — a pricing-threshold dodge, not a quality measurement [22]; the 272K figure is the entire input maximum GPT-5 launched with in August 2025 [15]. The same August 2026 engineering analysis — the write-up the pi observation comes from — recommends a quality-first profile of a 320K window compacting at 256K even on 1M routes, on the argument that "a model's native maximum is a capacity ceiling, not evidence of the optimal working-set size for a coding agent" — while itself calling that threshold "an engineering hypothesis to validate against your own workloads," not a measured optimum [22]. The vendor with the largest standard-priced window agrees with the skeptics against its own pricing page: Anthropic's September 2025 engineering guidance states that context "must be treated as a finite resource with diminishing marginal returns" and that degradation "emerges across all models," and the company ships compaction, context editing, and memory tooling as first-party API features [19]. A vendor does not build compaction primitives for a window that works as advertised.

The sharpest near-1M measurement on 2026 models comes from safety monitoring rather than the benchmark literature, and it lands on the same side. When Claude Opus 4.6, GPT-5.4, and Gemini 3.1 are used to flag dangerous actions in coding-agent transcripts (May 2026), they miss a subtly dangerous action 2x to 30x more often when it occurs after 800K tokens of benign activity than when it occurs alone; Opus 4.6 with thinking falls from 98.6% to 88% recall with 800K prepended, and from 99.7% to 69% when a blatant attack moves from a 100K to an 800K transcript [12]. This is precisely the regime the advertised window invites — very long, mostly-benign context with rare load-bearing tokens — measured on the Opus 4.6/GPT-5.4/Gemini 3.1 generation, at scale, and degrading badly.

The pricing signal, meanwhile, split in 2026 and created a live conflict. Anthropic's removal of the long-context premium (GA March 13, 2026) [20, 21] licensed a "just load everything" position. Five months later, the August 2026 compaction write-up still recommends a compacted working set — a 320K window compacting at 256K — even on 1M routes [22]; on that arithmetic, the measured-quality constraint did not move when the spend constraint did. No measured post-GA comparison — same workload, full window versus compacted working set, on Claude 4.6+ — appears in the cited literature, so this is an open empirical question, not a resolved one. What can be said: the burden of proof sits with full-window loading. Most broader or harder 1M evaluations report degradation; the exception on record is a single underpowered pilot in which one model, Gemini 3.1 Pro, completed an unbroken 5/5 three-hop series at each of 256K, 512K, and 1M [10] — evidence that parity is possible on one task for one model, not that it is typical.

The RAG-retirement question inherits the same answer. The 2024-generation evidence (legacy) found that, when resourced sufficiently, long context consistently beat retrieval on average — though retrieval kept a decisive cost advantage, which is why the same study proposes routing between the two — and that most models' retrieval-augmented performance decreased past a saturation point — 16K for gpt-4-turbo and claude-3-sonnet [24, 25]. On the 2025–26 generation, routing between retrieval and long context remains a live decision in the literature, on cost and position-sensitivity grounds [26]. No study in any generation recommends replacing retrieval with full-window stuffing; what changed is the reason to keep it — less about position curves, more about distractor sensitivity and non-lexical recall.

What this means in practice

The decision rules follow from the family-by-family evidence, and they are conditional on task and model generation.

1. Treat literal lookup at the full window as well-supported on the latest measured cohort — by a pilot, not a benchmark. Single-fact retrieval at 1M went 18/18 per model for Gemini 3.1 Pro, Claude Opus 4.7, and GPT-5.5 in the May 2026 exploratory pilot [10], and vendor-run needle tests report near-ceiling recall [14, 17]. Whole-document reference — "find the clause that says X" — is where the advertised window has its strongest evidence; a thresholded effective-length measurement at 1M still does not exist. 2. Keep retrieval and reranking wherever queries are non-lexical or distractor-rich. The 2023-era reason for chunking (position curves) has weakened; the 2025-era reason (non-literal retrieval collapsing at a few thousand tokens on the 2024–25 cohort, single distractors hurting) is measured [3, 13]. This folk wisdom survives with its mechanism replaced — and its stalest number is the one most worth re-testing on your own workload, since the thresholded record for latent-association retrieval ends at GPT-4.1, measured at 16K effective against 1M claimed [3]; no thresholded measurement on the GPT-5.5/Opus 4.7/Gemini 3.1 generation appears in the cited literature. 3. Retire middle-avoidance as a universal rule; keep it as a below-half-fill heuristic. Position effects are strongest when inputs occupy under ~50% of the window and fade near the limit (Aug 2025, ≤128K-window models); at high fill the dominant failure is not position but task family [8, 9]. 4. Start agent working-set budgets at roughly 256–320K as an engineering hypothesis, and validate on representative sessions. The anchors that cluster in that range are not quality measurements: pi's 272K default exists to stay inside OpenAI's short-context pricing tier [22, 15], the Claude Code early-compaction report is a single user-filed issue [23], and the WorkOS-style guidance calls its own threshold an engineering hypothesis to validate against your own workloads [22]. What gives the range weight as a starting point is that vendor engineering guidance and the long-transcript monitoring measurements point the same direction [19, 12]. Where a session must exceed the budget, put load-bearing state in durable storage rather than trusting the tokens past the budget line to be attended. 5. Choose the model by measured 1M behavior, not window size. The hardest common instrument shows a roughly threefold score spread across March 2026 flagships at 1M, and multi-hop cliffs are model-specific in the 512K–1M band [21, 10]. Two models with identical spec-sheet windows are not interchangeable at long length. 6. Date every number you inherit. A 2024 saturation curve, a 2023 position curve, and a 2025 effective-length table are claims about extinct model generations. The re-testing record here shows findings surviving, weakening, and inverting within eighteen months.

Evidence that would change these recommendations: a latent-association retrieval benchmark run on the GPT-5.5/Opus 4.7/Gemini 3.x generation (would settle whether rule 2 still binds on that cohort); a position sweep at ≥512K on a frontier model (rule 3); a controlled full-window-versus-compacted comparison on a post-GA workload (rule 4).

Conclusion

How much of an advertised million-token window functions as usable context? The question, as posed, outruns the measurements: an effective context length exists only relative to an explicit threshold applied to a score-versus-length curve, and no such measurement has been published at 1M for any model in the latest cohort. What the record supports, quantity by quantity: for literal retrieval, the GPT-5.5/Opus 4.7/Gemini 3.1 Pro cohort went 18/18 apiece in the one small pilot that tested the full million, and vendor-run needle tests agree — a real and recent shift from every predecessor generation, resting so far on thin samples. For non-literal retrieval requiring latent association, the thresholded evidence ends at GPT-4.1 — 16K effective against 1M claimed — with a few thousand tokens for the 2024–25 cohort before it [3]. For multi-needle and multi-hop work, the vendors' own tables and independent tracking report 1M scores between roughly 25% and 76% depending on model, with sharp model-specific cliffs past 512K — task scores at a length, not usable fractions of the window. For long-transcript vigilance, measurement on the Opus 4.6/GPT-5.4/Gemini 3.1 generation shows several-fold degradation by 800K. And the working-set budgets that shipped systems actually run — a quarter to a third of the advertised figure — are engineering starting points shaped by pricing tiers, single incident reports, and caution; treat them as hypotheses to validate on your own sessions, not as measured effective lengths.

The advertised number is not false. It is the answer to a different question — how much the API will accept, and increasingly, how much you will be billed for — and the folk wisdom built on 2023–24 measurements is neither all stale nor all sound. What is stale is any undated claim in either direction. The window stopped being a single spec and became a family of distinct quantities: what a given task scores at a given length depends on the task family, the model, and the date of the measurement — and all three now change fast enough that an effective-context claim without a threshold, a model name, and a date attached is not information.

Appendix: Limitations

Above roughly 256K tokens, most quantitative evidence is vendor-run: the 1M-scale MRCR curves originate in OpenAI's and Google's own publications, and the March 2026 cross-vendor comparison compiles vendor system cards run at differing reasoning-effort settings, so its per-model figures are indicative rather than directly comparable. The only fully independent 1M-scale academic measurement on the 2026 flagship generation is a self-described exploratory pilot — a single classical-Chinese corpus, few cells per condition, one query per cell — and its per-model figures are correspondingly noisy.

Two temporal gaps materially bound the conclusions. Collapse on non-literal retrieval requiring latent association — the earliest and largest failure mode — has been re-measured only as far as GPT-4.1 in the cited literature, not on the late-2025/2026 generation, so the possibility that it narrowed alongside literal retrieval cannot be excluded. Likewise, the original lost-in-the-middle position-sweep protocol has not been re-run at or near 1M on any model of the GPT-5.5/Opus 4.7/Gemini 3.1 Pro generation or its immediate predecessors; the "weakened, conditional" verdict rests on re-tests at 128K and below.

Finally, benchmark instruments disagree by construction: effective context length has no standard definition, and the truncation-protocol critique published in August 2026 shows that a portion of the "shorter context is better" literature measures experimental artifact rather than model behavior. Numbers here were preferred from designs that preserve the target content while varying length, but instrument-level heterogeneity limits how precisely any single effective-length figure should be read.