# How Language Models Learn What to Remember

> A compact memory becomes useful when a model learns how to write information into it and how to use it later. Following those two operations explains what survives compression—and what happens when the model keeps reading.

Published: 2026-10-28 · Lumisonde (https://lumisonde.com/reports/learned-context-memory/)
Citations JSON (per-claim receipts): https://lumisonde.com/reports/learned-context-memory/citations.json
Serial: Context Compaction (https://lumisonde.com/serials/agent-context-compaction/)
Role in this serial: Open the learned memory — Explains how trained writers, fitted caches and compatible readers turn compressed context into later computation, then distinguishes growing stores from bounded updates.
Previous in recommended reading order: https://lumisonde.com/reports/memory-compaction-implementation/

## The brief

**Question:** How does a language model turn what it reads into a smaller memory it can use later?

**Stake:** As documents and conversations grow, keeping every token available demands storage and computation. A smaller memory is useful only if it supports the later work that matters.

**Answer:** A model learns compact memory by learning what to write into a smaller numerical state and how to use that state later. [1, 2] That state may be continuous input vectors, cached attention records, or small parameter changes. [1, 2, 4] Feedback from reconstruction, prediction or answering teaches the writer or reader which information is useful; a better-trained reader can improve answers even when the stored bytes stay fixed. [1, 6] The practical test is which source distinctions remain recoverable for the future questions you care about, under the total storage and computation budget.

## Introduction

Imagine a passage describing a research project and giving its randomly assigned access code. After reading it, a model might need to explain the project's purpose, retrieve the code, or answer a question nobody anticipated. Keeping the whole passage available makes those jobs straightforward to specify: the relevant information is still in the input. Replacing it with a smaller memory raises a harder question: what must cross that smaller channel so a later computation can still succeed?

Learned context-memory systems make that channel part of the learning problem. A writer converts the source into a retained numerical state; a reader uses that state with a later question. Training supplies feedback when predictions made through this restricted path are wrong. Depending on the method, the retained object can be continuous input vectors or small changes to the computations inside the model. A reusable trained writer can create these passage-specific memories while the language model's original weights stay fixed. [1, 2]

The useful story connects three things: the source that was read, the state that survives, and the later behavior that state enables. Feedback can improve the writer, the reader, or an online update rule. Further reading then branches into another choice: add more stored pieces or rewrite a bounded state. Following these connections lets us distinguish information genuinely carried forward from information inferred by the reader—or recovered by returning to the source.

## Teach a smaller state to carry useful information

Start with a concrete writer. The In-context Autoencoder, or ICAE, appends special memory tokens to a passage. An encoder processes the passage and those tokens together; the output vectors at the memory-token positions become the passage's memory slots. Later, a language-model decoder receives the slots instead of the passage. These slots are continuous encoder-output vectors. [1]

During ICAE's autoencoding training, the decoder tries to reproduce the passage from those slots. Its predicted next-token probabilities are compared with the source tokens. If it assigns too little probability to the correct text, the resulting loss sends a gradient backward through the decoder and the slots to the trainable encoder. Optimization changes the encoder's low-rank adapter—a compact set of trainable weight additions—and the learned memory-token embeddings. The decoder's own weights remain frozen. A fixed reader can still transmit the error signal that teaches its writer. [1]

Two different objects therefore survive on different timescales. The trained encoder is reusable machinery for making memories. Its output slots are the memory of this particular passage. Once training is finished, the trained encoder creates a new passage's memory in a forward computation. [1]

ICAE also learns from text continuation, where the slots must help predict what follows the passage, and then from instruction examples, where the slots and a prompt must support a response. These targets ask the same bottleneck to support different behaviors: reproduction, continuation and answering. [1]

Its May 2024 revision reports Pile pretraining followed by prompt-with-context instruction tuning, with a default maximum source length of 512 tokens. [1]

The training objective makes retention consequential: information becomes useful to the learning process when carrying it through the memory helps reduce the chosen errors. Those errors supply the importance signal. Importance is therefore task-relative: predicting plausible prose and retaining an arbitrary access code can place different demands on memory.

Reconstruction is one way to teach the interface. Semantic-Anchor Compression starts from selected source-token positions, adds a learned anchor marker, and uses a bidirectional encoder so those positions can gather information from surrounding text. It retains their per-layer attention keys and values for a frozen decoder. The keys and values are the numerical records that later attention queries consult. The method trains with language modeling and QA without an autoencoding phase. [3]

The March 2026 v4 ablation tests this distinction with Llama-3.2-1B on MRQA at 15× compression. Language-modeling-only pretraining yields in-domain answer F1 of 54.95, compared with 51.73 after combined autoencoding and language-modeling pretraining. [3] The lesson is that learning a useful replacement for context and learning to reproduce context are related objectives whose alignment must be established. [3]

## Learning the writer is different from fitting the memory

The same distinction between machinery and passage-specific state applies when the memory lives in parameters. Doc-to-LoRA reads a document's internal activations with a hypernetwork: a neural network whose outputs are parameters for another network. It generates low-rank adapter matrices for a frozen target language model. With those matrices installed, later questions run through modified computations without receiving the document in their prompt. [2]

For an adapted weight matrix, the operation is

\[
W' = W + \alpha BA.
\]

Here, \(W\) is the original weight, \(A\) and \(B\) are the document-specific low-rank factors, and \(\alpha\) scales their contribution. The retained document memory is the generated factors. The hypernetwork that produced them is shared across documents. [2]

To train that writer, Doc-to-LoRA uses a context-conditioned teacher and an adapted student. The teacher sees the source and a query; the student sees the query while using the generated adapter. Training changes the hypernetwork so the student's response distribution approaches the teacher's. Across many documents and queries, it learns a reusable mapping from document activations to useful parameter changes. The fast document-time write is the product of this earlier training. [2]

Per-context fitting puts the optimization at a different point. Latent Context Compilation fits a temporary adapter for the current passage to produce a compact memory from restricted intermediate positions called a buffer. The buffer's final attention keys and values will survive as the memory. A strict attention mask allows buffer positions to read the source, while later query and response positions can read only the buffer and themselves. Its fitting tasks combine source repetition with unrelated instruction queries, using a full-context teacher's distributions as supervision. [4]

The temporary adapter's role ends with memory construction. The paper describes generation with that adapter disabled; after fitting, a final pass produces buffer-token keys and values, and the adapter is discarded. The surviving artifact is a compact KV cache, portable between compatible instances of the original frozen model. [4]

Together, these mechanisms separate two questions that the word “learning” often merges. When is optimization performed—across training documents or on this document? And what remains afterward—input-side state or parameter changes? A forward pass can produce parameter memory; gradient-based fitting can leave a cache. These examples separate how a memory is created from what is retained afterward. [2, 4]

## A later question reads through a learned interface

A later question can use the memory directly through the reader's computation. ICAE's decoder attends to memory vectors alongside the question. Doc-to-LoRA's decoder processes the question through its document-conditioned weights. In either case, the memory influences the answer through a compatible computation. [1, 2]

There can also be a separate read operation between persistent storage and the answering model. Memorilla, an October 2026 preprint, encodes document chunks once with a frozen embedding model and stores one embedding per chunk. For each question, a trained memory module conditions its slots on the query, attends over all stored document embeddings, and projects the result into 16 soft input vectors for a frozen Qwen3-8B decoder. [5]

The persistent collection is reused, but the 16-vector readout changes with the question. A query about the project's purpose and a query about its code can therefore ask the same store for different information. The 16-vector budget governs decoder input; the stored embedding matrix grows with the document collection. Memorilla trains the readout module to make that interface useful while leaving the encoder and language model fixed. [5]

The reader can also improve after the stored memory has stopped changing. In the October 2026 study *Evaluating Compressed Memory*, an experiment caches each QASPER evidence input as one 4,096-dimensional BF16 vector, or 8,192 bytes. [6] It then trains only the decoder's LoRA parameters, reusing the exact stored bytes without re-encoding the evidence or supplying its text. On the same 979 test questions, correct-document answer F1 rises from 14.21 to 21.29. [6]

Those inputs contain annotated evidence, so the experiment tests answering from supplied evidence. Within that setting, it demonstrates an important mechanism: changing how the reader uses a fixed memory can improve answers. The stored representation and the ability to access it are distinct parts of useful recall. [6]

## What, exactly, has been remembered?

Return to the illustrative access code. General knowledge can help a decoder reconstruct a conventional project description. It cannot identify a randomly assigned code from the project's purpose alone. If two passages differing only in that code produce indistinguishable retained states for an otherwise identical reader, the same later question cannot reliably recover both codes correctly. This is a logical constraint on the illustrative example.

Doc-to-LoRA shows that designated arbitrary digits can cross a learned parameter-memory bottleneck. Its synthetic needle-in-a-haystack experiment places a random four-digit number in text and asks the adapted Gemma-2-2B-it model to return it without the source. Retrieval stays close to perfect to about 40 chunks, or 40K tokens, beyond the base model's native 8K context. [2] The experiment trains on 32–256-token inputs and uses the same question during training and evaluation, with a task-specific ground-truth answer objective replacing generic QA distillation. [2] Longer inputs also receive larger composed adapters. It demonstrates length generalization for a learned exact-value task. [2]

The next question is how that memory handles more varied future uses. In Doc-to-LoRA's broader QA experiments, unseen test instances are answered using generated adapters, but internalizing unrelated passages can harm later SQuAD answering. The authors suggest that training on related document–query pairs encourages the model to assume subsequent questions concern its internalized document. Thus writing memory before seeing the test question and learning to handle unrestricted future questions are different achievements. [2]

ICAE provides another view of what shapes retention. With 128 slots, its pretrained Llama-7B reconstruction model nearly reproduces ordinary passages up to about 300 tokens, while smaller slot budgets deteriorate as passages lengthen. [1] At a fixed 512-token source length and 128 slots, its Table 3 reports BLEU of 99.3 for ordinary text but 0.2 for completely random text. [1]

Together, these findings connect retention to the training task, the source's predictability and the later question. A specifically trained writer can preserve exact digits. A representation that supports one kind of question can still fail another. A decoder can also draw plausible material from its prior knowledge.

The compressed-memory audit tests this last issue through distinct comparisons. In its released xRAG experiments, correct retrieved passages outperform donor passages: this contrast tests the contribution of the source's content. Donor passages also produce some gains over no document. Another control uses separately trained constant-input readers. Asking whether an answer changes appropriately when the source changes is more informative about retained content than asking only whether adding a vector improves an answer score. [6]

Sometimes the compact state has a different job: help the model decide where to look again. The June 2026 study *End-to-End Context Compression at Scale* jointly trains an encoder and decoder to use compressed latent inputs. Its agent variant keeps numbered 512-token original chunks and lets the model call `EXPAND(i)` to retrieve a chosen chunk's text. [7] On eight RULER retrieval tasks at 16K source length, the 16×-compressed 0.6B-encoder/4B-decoder model's average score rises from 70.18 to 89.91 with the expansion loop. [7]

Here the compact view gives broad access to the material, and the source store supplies precise detail on demand. That is useful memory-assisted navigation. Its success belongs to the combination of compressed context, selection behavior and retained source access; it answers a different question from exact recall after the source has been removed. [7]

## Keep reading: enlarge the store or rewrite it

One passage can become a reusable memory. A stream of passages requires a policy for what new writing does to old state.

AutoCompressors accumulate learned summary vectors from earlier segments. Each new segment receives the concatenated summaries of all preceding segments, and next-token prediction on later text teaches the model what those summaries should carry. If each segment contributes \(\kappa\) vectors, the memory before segment \(i\) contains \((i-1)\kappa\) vectors. The method compresses the history while allowing its retained length to grow. Its November 2023 v2 paper also acknowledges that attention remains quadratic as summaries accumulate. [8]

Doc-to-LoRA grows parameter memory instead: it generates an adapter for each chunk and concatenates their factors along the rank dimension. With \(K\) chunks and per-chunk rank \(r\), the composed rank is \(rK\). The strong magic-number length result therefore uses increasing memory capacity as well as a learned write rule. [2]

A bounded memory changes the problem from adding records to updating a state. Titans illustrates this with a neural associative-memory module. Incoming activations are projected into keys and values: a key supplies the cue for the memory function's prediction, and its associated value supplies the target. The difference between prediction and target produces an inner-loop gradient update to the contextual memory's weights. Those changed weights are the retained state. Momentum carries update information forward; input-dependent weight decay controls how much previous state remains. Outer training learns the surrounding architecture, including the projections that define these associations. [9]

A later projected query consults that changed memory function without updating its weights for the lookup. The function produces activations that feed the surrounding sequence model; Titans combines this memory with mechanisms such as local attention and input-independent learned memory. This is a designed sequence architecture whose contextual memory changes online. [9]

A bounded state can carry selected information through many updates. A 2024 Recurrent Memory Transformer study augments pretrained BERT with ten memory vectors and trains through a curriculum of up to seven segments. Its synthetic classification tasks require retaining one fact, detecting one fact among distractors, or relating two facts. Tests extend to 4,096 segments—about two million tokens—with strong sparse-fact retrieval generalization; reasoning is harder than simple detection. [10]

The distinction is between the duration of retention and the amount of information that must survive. Carrying one task-relevant fact through thousands of segments can fit a bounded state. Preserving every new identifier, relation and correction poses an increasing information load. Growing memories and recurrent memories address that load differently: one allocates more retained state, while the other must learn which distinctions to carry into the next update. Comparing them therefore requires counting both the state that remains and the work needed to write and read it.

## Compactness belongs to a workload

The surviving object's layout matters as much as its advertised length. A soft input vector occupies one embedding-width array; a cached position can require both keys and values across many layers; a low-rank adapter stores matrix factors. A short query-time readout can coexist with a large persistent store. Slot counts, cached positions and adapter ranks are consequently different units of storage. [1, 2, 4, 5]

The computational account separates shared training, encoding or fitting each source, and the question-dependent read and decoder work needed for each later use. Where answering can return to original text, its storage and access also belong in the account. The quantities to request are total retained bytes and the work spent writing and reading them. From these mechanisms follows a workload-level accounting rule: a write paid once can be amortized over repeated questions only while its artifact remains reusable. Total cost depends on all of these components.

## Conclusion

A language model learns compact context memory by making later computation depend on a restricted retained state, then learning a writer, reader or update rule that makes that dependence useful. The memory can enter as vectors, cached attention state or parameter changes. Its value lies in the distinctions a later reader can recover from it.

These mechanisms reveal several ways to make memory useful. A trained writer can generate new parameter memory in a forward pass. Optimization can leave only a portable cache. A better reader can extract more from unchanged bytes. A small recurrent state can carry selected facts over a very long stream, while a compressed document memory can grow as reading continues.

The next design question is therefore concrete: which distinctions must remain answerable, for which future uses, under which storage and computation budget? Answering it connects how memory is taught to what that memory can actually do.

## Limitations

This account uses paper-reported mechanisms and experiments available through October 26, 2026; it does not establish implementation equivalence or independently reproduce their results. References identify the paper versions where known. The Doc-to-LoRA full text consulted was available through Hugging Face, but its exact revision was not established.

The studies use different models, training data, capacities and evaluation procedures, so their results do not form a matched performance or lifetime-cost ranking. Exact-value retrieval, source reconstruction, natural QA and sparse-fact retention test different demands on memory. Controlled preservation of contradictions and corrections, reliability under unexpected future questions, cross-model memory transfer, and retention under indefinitely changing streams remain substantive open questions.

## References

[1] Ge et al. *In-context Autoencoder for Context Compression in a Large Language Model*. arXiv:2307.06945v4, May 8, 2024. Sections 2–3, Figures 4–5 and Table 3. — https://arxiv.org/abs/2307.06945v4
[2] *Doc-to-LoRA: Learning to Instantly Internalize Contexts*. arXiv:2602.15902, February 2026; full text consulted through Hugging Face, exact revision not established. Sections 3–6 and Appendices A and C.4. — https://huggingface.co/papers/2602.15902
[3] Liu et al. *Autoencoding-Free Context Compression for LLMs via Contextual Semantic Anchors*. arXiv:2510.08907v4, March 11, 2026. Sections 3–4 and Table 5. — https://arxiv.org/html/2510.08907v4
[4] *Latent Context Compilation: Distilling Long Context into Compact Portable Memory*. arXiv:2602.21221v1, 2026. Sections 3.1–3.5. — https://arxiv.org/html/2602.21221v1
[5] *Memorilla: Latent Semantic Memory for LLMs*. arXiv:2610.21375v1, October 15, 2026. Sections 2.1–2.6. — https://arxiv.org/html/2610.21375v1
[6] Oh et al. *Evaluating Compressed Memory: Dependence, Utility, and Content Benefit*. arXiv:2610.19373v1, October 14, 2026. Sections 3–8 and Appendix D, Tables 11–12. — https://arxiv.org/html/2610.19373v1
[7] *End-to-End Context Compression at Scale*. arXiv:2606.09659v1, June 2026. Sections 3–4 and 7; Table 33. — https://arxiv.org/html/2606.09659v1
[8] Chevalier et al. *Adapting Language Models to Compress Contexts*. arXiv:2305.14788v2, November 4, 2023; EMNLP 2023. Section 3 and Limitations. — https://arxiv.org/abs/2305.14788v2
[9] *Titans: Learning to Memorize at Test Time*. arXiv:2501.00663v1; the v1 header is dated December 31, 2024. Sections 3–4, especially Equations 11–15. — https://arxiv.org/abs/2501.00663v1
[10] Bulatov et al. *Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory*. AAAI 2024. Memorization Tasks, Learning Memory Operations and Figures 4–5. — https://ojs.aaai.org/index.php/AAAI/article/view/29722/31239

---
Every [N] marker above is verifiable: fetch the citations JSON, match the marker id, and check the extraction (summary and key facts) plus placements against the source URL. Verification guide: https://lumisonde.com/for-agents/