The KV Cache Became Inventory — and Almost Nobody Charges Rent Yet

Why anyone would build a warehouse for disposable data

NVIDIA is building petabytes of purpose-built flash for data that is, by design, disposable. The CMX context memory storage platform establishes what the company calls a G3.5 layer, an Ethernet-attached flash tier optimized specifically for KV cache, providing petabytes of shared capacity per GPU pod so that long-context workloads can retain history after it has been evicted from GPU and host memory [1]. The scale is not gestural. A Vera Rubin SuperPod pairs 1,152 GPUs with sixty-four BlueField-4 data processors, each fronting 150 TB of NVMe — some 9,600 TB of flash of which an NVIDIA networking manager says plainly: "The storage infrastructure's sole purpose is to serve inference context memory." [2]

NVIDIA's own framing names the strangeness of this. KV cache, it writes, is a class of data defined by a duality: critical for performance yet inherently ephemeral [1]. Every byte in those racks can be recomputed from the token history that produced it — the prompt plus everything generated since — given the same weights, positions and inference configuration. So why fund a warehouse for something you could always rebuild?

Because rebuilding is the expensive part, and the thing has grown too large to keep where it used to live. A transformer serving a request does two different jobs. Prefill reads the whole prompt at once and computes, for every token and every attention head, a key and a value vector. Decode then emits one token at a time, each new token attending over every key and value already computed. The KV cache is where those vectors are kept so that decode never has to recompute them. Its size is proportional to batch size, sequence length, layers, key-value heads and head dimension, twice over because keys and values are stored separately [5]; the byte count follows by multiplying through by the precision, in bytes per element [40, 13]. Two of those terms are levers rather than constants. Precision is the one a deployment sets directly [40]. The other is the key-value-head count, which is not the head count: Llama-3-70B uses eight key-value heads for sixty-four query heads, an eightfold reduction against multi-head sizing [13, 41]. Every term multiplies and none of them shrinks on its own, so the cache grows linearly in both users and context.

Linear growth from a large base gets somewhere alarming quickly. On NVIDIA's own numbers, Llama 3 70B in half precision needs roughly 140 GB for its weights, while a 128k-token context for a single user consumes about 40 GB of KV cache, "scaling linearly with the number of users" [3]. Four users and the inventory exceeds the goods. Push the window further and the inversion is unambiguous: running LLaMA-3.1-8B with a one-million-token context requires storing over 120 GB of KV cache, past the memory capacity of most GPUs — for an eight-billion-parameter model whose weights fit in a fraction of that [4]. At the top end, one inference-scaling study puts Llama-3.1-405B at roughly 1.05 MB of cache per token in FP16, reasoning from its roughly 126 layers [6] — about twice what the same per-layer arithmetic gives for the model's published grouped-query configuration of eight key-value heads at a head dimension of 128 [13, 41]. The study does not state the head count behind its figure, so take the order of magnitude rather than the digits.

Prefill manufactures, the KV cache is inventory, and decode picks from stock. A cache hit ships from the warehouse instead of restarting the factory. When inventory runs this large relative to the product, the binding constraint stops being factory throughput and becomes shelf space.

The constraint moved from the factory to the warehouse

A 2026 survey of storage and memory for LLM inference puts a number on where that constraint sits. It defines a crossover batch size, B*, as the batch at which KV-cache traffic overtakes weight traffic during decode; below it you are weight-bound, above it you are cache-bound. For Llama-3 70B with FP16 weights at a 4,096-token sequence, B* sits at approximately 108 [7] — a comfortable regime where compressing weights is the higher-return move. Quantize the weights to INT4 and B* falls to about 27. Extend the sequence to 131,072 tokens and B* drops below one: a single sequence, one user, no batching at all, is already dominated by KV-cache I/O [7].

Under those conditions, buying arithmetic does not buy throughput. Xiaoyu Ma and David Patterson open their hardware paper, accepted for publication by IEEE Computer in 2026, with the flat sentence "Large language model (LLM) inference is a crisis" and argue that, exacerbated by recent AI trends, the primary challenges are memory and interconnect rather than compute. Their diagnosis is architectural rather than incidental: the current AI hardware philosophy — full-reticle dies with high FLOPS, many stacks of high-bandwidth memory, bandwidth-optimized interconnect — is, they write, "a mismatch to LLM Decode inference" [8]. The trend line behind that judgement is long-running. Reporting on the same authors, SDxCentral notes that NVIDIA GPU 64-bit FLOPS rose roughly eightyfold between 2012 and 2022 while bandwidth grew only about seventeenfold, and quotes their expectation that the gap will continue expanding [9]. TrendForce frames the same divergence from the memory side: compute grows about three times every two years, memory bandwidth roughly 1.6 times [34].

In the conditions tested — these model sizes, these sequence lengths, these precisions — decode is capacity- and bandwidth-bound, and adding compute changes little. That is a measured property of current hardware ratios running current architectures rather than a law of nature, and the papers say so.

The pressure is also rising faster than context-window announcements suggest, because reasoning models manufacture inventory while they work. An empirical study of inference scaling on H200 nodes finds that reasoning traces routinely exceed 10,000 output tokens per sequence, against roughly 500 for ordinary chat, and that the cache accordingly grows linearly through decode until it exhausts the per-GPU high-bandwidth memory, or HBM, that holds it [6]. The same work names a capacity trap for mid-sized models, where maximizing GPU occupancy for throughput fights directly with memory availability, producing stranded capacity and premature request throttling despite low compute utilization [6]. Growth used to arrive with the prompt. Now it arrives with the answer.

Two moves remain: pack the inventory smaller, or move it somewhere cheaper.

Packing it smaller, and where the floor is

Two distinct operations get bundled under compression. Quantization keeps every cached token but stores each vector in fewer bits. Eviction keeps full precision but discards tokens judged unlikely to matter. They compose, and they fail differently.

Quantization arrived first, and its returns are real. KIVI, presented at ICML 2024, quantizes the cache to two bits with no fine-tuning and reports 2.6 times lower peak memory usage including model weights on Llama, Falcon and Mistral, allowing up to four times the batch size and lifting throughput by 2.35 to 3.47 times [10]. Its more durable contribution is structural rather than numerical. The key cache and the value cache are not the same kind of object: "the key cache should be quantized per-channel, i.e., group elements along the channel dimension and quantize them together. In contrast, the value cache should be quantized per-token" [10]. KVQuant, at NeurIPS the same year, pushes to three bits with under 0.1 perplexity degradation on Wikitext-2 and C4, taking LLaMA-7B's cache from 64.0 GB to 12.0 GB and enabling a one-million-token context on a single A100-80GB [11].

Architecture pulls the other lever, and it pulls it before any bits are dropped. Multi-head attention stores a key and a value per head; grouped-query attention shares one key-value pair across a group of query heads; multi-head latent attention instead projects keys and values down to a single low-rank latent vector and caches only that, reconstructing them at use [37]. TransMLA proves the arrangement is not merely cheaper: for the same KV-cache size, the expressiveness of latent attention is greater than that of grouped-query attention, and any grouped-query model can be transformed into an equivalent latent-attention one [12]. The gap this opens is large enough to be a systems problem in its own right. DeepSeek-V3 under latent attention requires 1,152 bytes per token per layer, against 65,536 under the multi-head-equivalent sizing that general-purpose serving frameworks still assume — a 57× memory penalty imposed by the framework rather than the model [13].

Recent work keeps finding headroom. MosaicKV compresses along both the sequence and the channel dimension at once, reporting a 3× reduction in memory with 1.76% average accuracy loss on LongBench and RULER, and 7.3× higher throughput than an uncompressed baseline on an H800 [14].

But the ceiling is not a matter of waiting for a cleverer algorithm. A 2026 theoretical result characterizes the minimax risk of KV-cache compression in terms of the intrinsic compressibility of a cache, showing when accurate compression is possible and when it is not. Worst-case compression was already known to be impossible, and the achievable ratio turns out to be a property of the particular cache rather than of the method applied to it [15]. Empirical work reads the same way from the other end. A controlled study of quantized cache archives on GPT-2 measures a sharp asymmetry between the two halves of the cache: quantizing keys is roughly 4× more damaging than quantizing values, a finding the authors then use to allocate bits unevenly [16]. Compression budgets, in other words, are set by the structure of what you are compressing.

Compression does not only subtract. Eviction has been treated as an approximation of full-cache inference — the question being how little you lose. One 2026 method inverts that premise. Its authors argue that full-cache attention is not always optimal, because in long contexts irrelevant tokens dilute attention away from useful evidence; a learned, globally calibrated eviction policy, in which tokens from different layers and heads compete for a single memory budget, substantially reduces KV memory while matching or surpassing full-cache inference on long-context language, vision-language and multi-turn benchmarks [17]. Attention dilution, on this account, is a real cost of keeping everything, and eviction can be a mechanism for improving reasoning rather than only a tax on it.

The freed memory can also be spent rather than banked. Inference-time hyper-scaling compresses the cache in order to generate more tokens inside the same compute budget, using a sparsification method that reaches eight-fold compression after about a thousand training steps. Applied to Qwen-R1 32B at an equivalent number of memory reads, it improves AIME 24 by 12.0 points, GPQA by 8.6 and LiveCodeBench by 9.7 [18]. Carrying less inventory made the product better.

Moving it instead: the cache becomes addressable infrastructure

The second move runs alongside the first rather than after it: shrink what you keep, relocate what you cannot shrink.

The chain starts with allocation. Before PagedAttention, a serving system reserved a contiguous block of memory sized for the longest sequence a request might produce — the same 2,048-token reservation whether the user sent fifty tokens or seven hundred — and the surplus was stranded [38]. Paging the cache into fixed-size blocks, sixteen tokens per block by default on CUDA, allocated on demand, removes the external fragmentation and leaves only the partly-filled last block of each sequence [20, 38].

Blocks are what make the cache addressable, and addressing is what makes it reusable. vLLM identifies each cached block by a hash over the tokens inside it together with the tokens of the prefix before it, so that a new request sharing a prefix with an old one finds the computed blocks already there; the default hash has been sha256 since v0.11, chosen to address collision risk [19]. A shared cache is also a shared side channel, so the same documentation offers an optional per-request salt, injected into the hash of the first block, which isolates cache reuse and prevents timing attacks in multi-tenant environments [19]. That is the moment a buffer becomes a namespace.

Once blocks are addressable they can also be located. In a multi-instance deployment each replica holds a different cache, so a cache-blind load balancer scatters requests that share a prefix and destroys the locality it needs. Routing to the instance whose cache already holds the request's prefix is now a first-class scheduling primitive [21]. On a cluster of eight vLLM pods across sixteen H100s, running a business workload of many customers each carrying a distinct system prompt, precise prefix-cache-aware scheduling reached a P90 time-to-first-token of 0.542 seconds at 8,730 output tokens per second [22, 39]; in the same table, from the serving project's own benchmark, the approximate cache-aware scheduler took over 31 seconds at that percentile and the cache-blind schedulers over 90, on identical hardware [22]. A production account from a different vendor puts the mechanism more simply: without prefix-aware routing roughly one request in four lands on an instance with the prefix already warm, and with it that flips to three in four, in a workload where about seven requests in ten share a system prompt [20].

Then the cache is tiered. NVIDIA's Dynamo documentation offers three interchangeable backends for extending effective cache capacity beyond GPU memory into CPU RAM and disk [23]. Research systems formalize the hierarchy further, into six tiers running from GPU high-bandwidth memory through CPU DRAM, CXL memory, NVMe with direct storage access, RDMA network and parallel filesystems, each with its own latency and cost per gigabyte-hour [13]. CMX's G3.5 flash is one more rung on that ladder, sized at the pod rather than the node [1].

The hardware bet points the same way. Patterson and Ma's leading proposal is High Bandwidth Flash: a stack offering 512 GB of capacity at 1,638 GB/s read bandwidth under 80 W, against an HBM4-6400 stack's 48 GB at the same bandwidth — roughly 10× the memory capacity with HBM-like bandwidth, at over 20.5 GB/s per watt, paying for it in read latency measured in thousands of nanoseconds rather than tens [8]. Capacity-first memory answers a capacity-bound workload, and it remains a research direction rather than a shipping product.

The quiet consequence is that the cache stops being ephemeral in practice even where it remains ephemeral in principle. Retention windows are lengthening from minutes into hours: Anthropic's cache defaults to five minutes with a one-hour tier sold at twice the base input price [27, 28]. One blog analysis of the CES 2026 platform — four BlueField-4 processors fronting about 600 TB per appliance — reaches further on the strength of its author's own habit of resuming chat sessions from days or weeks earlier, arguing that cache therefore cannot simply live in GPU memory and disappear but must be offloaded, stored and retrieved when a session resumes [24]: "KV cache is no longer a temporary artifact. It's becoming persistent state."

An asset you keep, route to, and build racks for is an asset someone will eventually price.

And the price signal points the other way

Someone has, and along an axis that at first looks like the wrong one.

DeepSeek's published rates split the input price by cache state and nothing else. V4-Flash bills input at $0.14 per million tokens on a cache miss and $0.0028 on a hit; V4-Pro bills $0.435 against $0.003625 [26]. That is a spread of fifty times on the smaller model and roughly a hundred and twenty on the larger — differences of a kind normally reserved for choosing a different vendor, applied here to whether the provider had seen your prefix recently.

Anthropic's structure prices the same idea with an explicit write premium. For Claude Fable 5, base input is $10 per million tokens and output $50, while cache hits and refreshes bill at $1; writing to the cache costs $12.50 for the five-minute tier and $20 for the one-hour tier, the longer time-to-live being available at twice the base input price [27, 28]. The arithmetic of that premium is not close. A five-minute write costs 1.25× the standard input rate and a read costs 0.1× — a ninety percent discount [29] — so the premium is recovered well before the first full re-read. Caching here is not an optimization to schedule for later; it is the default that costs money to decline.

The second half of the move is what disappeared. Anthropic's pricing documentation now states that Claude 4.6 and later models include the full 1M token context window at standard pricing, with caching and batch discounts applying across the whole window [27]. A depth penalty existed; one write-up describes flat-rate long-context pricing as having removed it [35]. Third-party calculators have not caught up, and still list a doubled input rate past two hundred thousand tokens for Anthropic [36]. For Google the same listing is not stale. Nothing in the public record has Google withdrawing a depth tier: the same calculator puts Gemini Pro at twice input past that threshold [36], the flat-rate write-up names Gemini as the most cited example of length-based pricing [35], and the caching survey notes that some providers still tier above 200K tokens [30]. The bill is being cut on whether context is reused rather than how long it is — everywhere except Google.

A cache hit spares the provider a prefill: the tokens were already read and their keys and values already computed, which is precisely the work prefix caching is designed to skip [19]. So the discount is, mechanically, a discount on arithmetic not performed. It is not a discount on memory occupied, because occupancy is not what is being measured. The physical picture from the preceding sections is the opposite one: the expensive, scarce, capital-intensive resource is the shelf space, and the shelf space is exactly what no line item counts.

One vendor does count it. Google charges a per-hour cache storage fee in addition to its cache-hit rate [30]; one cost guide describes the same arrangement, cached content billed at about a tenth of base input plus an hourly storage charge [31]. DeepSeek's own rate card has no counterpart: it prices input by cache state alone, a hit rate against a miss rate, with no write or storage line anywhere on the page [26]. OpenAI enters this only at second hand: the same aggregator reports that OpenAI, DeepSeek and others typically charge cache-hit pricing with no write or storage fee, and no first-party rate card is cited here to confirm it [30]. The exception shows the rule is a choice rather than a technical necessity — and it makes Google the exception twice over: it meters the shelf, and it still prices depth [30, 36].

What operators capture from the gap shows up only in the accounts they publish themselves. Deriv reports on its own engineering blog that its production assistant reached a prompt-cache hit rate in the mid-eighties and cut input-token costs by roughly three-quarters, without changing the model or removing context [32]. A second write-up, at second hand, describes ProjectDiscovery moving dynamic working memory out of its system prompt and into a trailing message, taking its hit rate from single digits to the mid-eighties [33]. Neither is a rate anyone can plan against; what they establish is only that the achievable gap is large and that prompt layout, not model choice, is what moves it.

Meanwhile the resource nobody bills is getting scarcer. Memory capacity is being prioritized for high-bandwidth and high-end server parts, squeezing supply elsewhere to the point that low-end smartphones are expected to ship with 4GB again in 2026 [34], and SK hynix has warned of elongated lead times stretching out to 2028 [9]. These are trade-press and market-research estimates rather than audited figures, and they should be read as direction rather than magnitude.

Nothing in the public record ties any provider's discount schedule to its measured memory cost; the correspondence argued here runs from the mechanism, not from an operator's ledger. No source prices a gigabyte-hour of cache residency in production. And the operator hit rates above are first-party and self-selected — teams that fixed their caching wrote about it.

What has a clock on it

Three things would put a price on storage, and each is observable.

The first is metering. If cache residency or time-to-live becomes a line item, the asymmetry closes directly, and Google's per-hour storage fee is the leading indicator to watch [30]. The second is capital. CMX-class tiers are expected to begin arriving in customer datacenters in the second half of 2026 [25]; racks of flash dedicated to context put a capital cost on occupancy that someone eventually has to recover, and the vendor's own argument for the tier is that power availability is the primary constraint on scaling AI factories [1]. The third is what "cheap to hold" comes to mean. Capacity-first memory such as High Bandwidth Flash would reset what holding costs at all, and it is not shipping [8].

Until one of those lands, the gap between what is scarce and what is billed is an operator's advantage rather than a grievance, and the evidence above says where to take it. Meter the hit rate first: both published accounts of large savings report one, and neither reports a model change [32, 33]. Fix prefix layout before buying capacity, because layout rather than model choice is what moved those numbers — one team went from single digits to the mid-eighties by relocating volatile content out of a system prompt [33]. Match cache lifetime to call cadence: the longer time-to-live carries a real premium, and a five-minute write pays for itself before the first full re-read [27, 28, 29]. Route before you tier, since on the benchmark above the scheduling choice alone was the difference between sub-second and minute-plus P90 latency on identical hardware [22]. And price the spread, because when a hit and a miss differ by fifty times the hit rate has stopped being a performance metric and become a pricing term [26].

There is one result here that deserves to outlast the pricing question. If the cache is inventory, the most useful finding of 2026 is that carrying less of it can make the product better — that dropping tokens can sharpen attention rather than blur it [17], and that memory freed by compression buys accuracy back when spent on reasoning [18]. The cheapest cache is still the one you never had to keep.