What Should an AI System Remember?

In Where an LLM System’s Knowledge Lives—and How to Change It Safely, we explained that an LLM system has no single knowledge store: information can reside in model parameters, transient context, or persistent external stores. Here we focus on one part of that picture: when a conversation ends, what is worth retaining to improve the next interaction—and under what conditions should that memory be used again?

For engineers, the central object should be a retained observation with a use contract: what was said, by whom, about which situation, when it applied, and what later use is allowed. This is a design recommendation, not a claim that every memory system needs the same schema. Its purpose is to keep distinct decisions distinct. Storing an observation is not establishing its truth; accepting it as true is not making it current; making it current is not authorizing its use in every answer.

The practical goal is useful continuity: less unnecessary repetition and better decisions without unwanted carryover. Start by specifying that improvement, then work backward to what must survive.

Begin with the answer memory should improve

Consider an illustrative coding assistant. During an incident, a user asks for terse updates for the internal operations channel. Retaining that request could help when the user resumes the same incident tomorrow. It does not yet justify a permanent profile saying the user always prefers terse writing, much less using that profile when drafting a sensitive message to a customer.

The admission question—the decision to retain information—is therefore concrete: what future task would be worse if this detail were unavailable?

Illustrative admission reasons proposed here, not measured results. These are not a taxonomy that a model must populate from every conversation.

There is a real distinction between retaining something and benefiting from it. In interviews reported in From Storage to Interpretation, a participant described an assistant acknowledging a remembered development-tool version while continuing to give guidance relevant to older versions.[1] The same participant described a request about salt balance in a particular recipe being interpreted as a lasting preference for future recipes.[1] The first is a failure to use relevant context; the second is a failure to preserve its boundary.

For each candidate, write down the anticipated improvement and the smallest scope in which it is justified. Keep incident-specific instructions with the incident. Promote a preference to a broader default only when there is a reason to believe the broader interpretation is intended; asking the user is an available option. If no plausible future use can be named, leave the detail in the current interaction rather than manufacturing a durable personal fact.

Smallest scope does not mean shortest possible text. LongMemEval's memory-granularity experiments found that compressing conversational rounds into individual user facts harmed overall performance through information loss, even while helping multi-session reasoning.[2] Its results support choosing a representation for the questions it must answer, not assuming that removing more context always improves memory.

Preserve the observation, not just its extracted label

A proposed memory should make it possible to distinguish the user's statement from the system's interpretation. In the incident example, the observation is a request for terse internal updates during that incident. A general preference for brevity is an inference. Treating the inference as though the user explicitly established it changes the meaning of the conversation.

A practical memory record could carry the following information:

These fields are a proposed contract, not a validated universal format. Content kind, derivation, and certainty are separate properties, not mutually exclusive statuses. They need not all appear as prose in a prompt. Stored conditions inform permission decisions; the application's control path should enforce product policy, including checks of current purpose and audience before use. What matters is that the distinctions survive whatever representation the product uses.

An explicit user statement also has limits. A user can authoritatively express a preference without making an assertion about another person verified, or turning pasted text into a trusted operational instruction. Keep the content's origin separate from the authority it would need to control future behavior.

The security consequence is not hypothetical: the MINJA study demonstrated memory injection through queries and observations of outputs, without assuming direct access to modify the memory bank, so that malicious records could influence later victim queries in the tested systems.[3] The engineering implication is that a successful write through the normal interaction path must not itself confer instruction authority. A memory describing an earlier interaction should remain evidence about that interaction, not become a command privileged over the present task.

Preserving origin gives the system something to check. The actual permission decision still belongs in the product's control path: decide which kinds of remembered material may influence which operations, and test whether those boundaries hold.

Make every reuse earn its place

Once a memory has been admitted, a later query creates a different question: is using it appropriate here? Similarity to the query can nominate a memory for consideration. It cannot, by itself, settle audience, purpose, or permission.

BenchPreS tests whether persistent user preferences are appropriately applied or suppressed in communication contexts involving other people and social or institutional norms.[4] The authors report that stronger preference adherence was associated with more over-application, and that neither reasoning capability nor prompt-based defenses fully resolved the problem in their evaluation.[4] An assistant can therefore succeed at remembering a preference while failing at deciding when to honor it.

For the incident assistant, the internal terse-writing instruction should not silently determine the tone of a customer-facing explanation. If the user explicitly requests the same style for that new audience, the current request supplies a new basis for using it. Without that basis, the assistant can use the current task's requirements, or ask a targeted question if the choice materially affects the result.

Scope is also a user-control issue, not merely a classifier feature. In the interview study, participants described separating topics into different chats and selectively copying context between chats, while others deliberately kept a single conversation for continuity across topics.[1] A product should not assume either universal separation or universal carryover from the existence of past information.

My recommendation is to make the narrowest useful boundary the default and let users broaden it deliberately. Offer a way to keep something within a project, exclude it from later personalization, or remove it. For sensitive details, product policy should govern permitted purposes and audiences even when use would improve an answer; ask the user a targeted question when intended use is materially ambiguous. A user might want a personal constraint respected within private planning without having that constraint disclosed in text intended for somebody else. The design should allow restricted use where appropriate, not force a choice between global reuse and retaining nothing.

Inspection should likewise reach beyond a list of stored facts. Let the user see which remembered observation affected a consequential answer, why it was eligible, and how to change that future behavior. The control the user needs is over the consequence of remembering, not only the existence of a record.

Separate current state, history, and deletion

An admitted memory is a candidate for future use, not a permanent default. Return to the incident example: when the incident closes, the terse-update request can stop governing new drafts. Its historical role may still matter if the user later asks why earlier reports took that form. The design should distinguish that historical question from a request to produce today's report.

This suggests separating a record's retention from its eligibility as current guidance. A superseded observation may remain available for an authorized historical query while no longer appearing among current defaults. An uncertain observation may require confirmation. A withdrawn observation should follow the product's removal policy rather than merely become less prominent in search.

There is empirical reason to test this transition explicitly. In the Supersede study, bounded, self-maintained memory performed worse than full context for gpt-5.4 on the knowledge-update subset of LongMemEval.[5] That comparison concerns maintaining updated facts, not simply locating relevant text.[5] For a product team, the corresponding test is whether a correction changes the next answer and any dependent defaults, not just whether the new statement was saved.

Use expiry and confirmation rules that reflect the kind of information involved. A temporary task instruction can end with the task; an uncertain inference can remain unconfirmed; a consequential assumption can be checked when it matters. Do not interpret age alone as evidence that a memory is false. Equally, do not interpret the absence of a correction as proof that it remains true. These are policy choices to validate in the intended workflow.

Deletion requires a separate promise. OpenAI's Memory FAQ distinguishes its memory summary from the broader context used for personalization and says the summary need not contain everything remembered.[6] It also says fully removing information can require deleting the sources where it appears, and that retaining chat history can allow memories to be created again after memory is re-enabled.[6] The general design lesson is to inventory the paths by which a supposedly removed detail can return.

Specify which records, summaries, indexes, and source histories a removal request affects. If some data remains under a separate retention policy, explain that distinction rather than presenting removal from personalization as erasure everywhere. Then test the behavior: after removal, can the detail still influence a response or be reconstructed from a surviving input?

Test useful continuity and harmful carryover separately

The admission policy should earn its complexity in evaluation. Recall is a component of that evaluation, not its final objective. LongMemEval measures information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention through questions over interaction histories.[2] BenchPreS separately measures appropriate preference application and misapplication across communication contexts.[4] These are complementary targets: finding the right information and deciding whether to apply it.

For a product evaluation, compare the proposed policy with a memory-free baseline on the same later tasks, holding the answer model and task instructions fixed. Where permission and feasibility allow, add a full-history comparison to reveal information lost during extraction. Track latency and total processing cost separately from answer quality so an efficiency gain does not masquerade as a benefit to the answer itself.

Build cases in which memory should help and matched cases in which the same detail should remain silent. For the incident assistant, test a resumed internal update, an unrelated customer message, a user correction to the style, and a request to remove the preference. Include ambiguous statements, pasted third-party content, changed project conditions, and questions whose premise assumes an obsolete state. Label the intended behavior before scoring the output.

The product-level outcomes should include avoided repetition, successful task completion, inappropriate disclosures or carryover, and the work required to correct the assistant. Ask users whether the remembered context improved the result and whether its use was expected. Keep the help and harm results separate: an average that combines easier continuation with unwanted personalization obscures the decision the team actually faces.

When a case fails, locate the failure before changing the policy. Was useful information never admitted? Was its scope lost during extraction? Was a valid memory applied in the wrong context? Did a correction fail to displace an old default? Each failure calls for a different change. Increasing retention indiscriminately is not a diagnosis.

Conclusion

An AI system should remember information that can improve a future interaction, while preserving the basis and boundaries for that improvement. The durable unit is not simply a fact about the user. It is an observation whose origin, scope, currency, and permitted uses remain available when the system considers acting on it.

Start with a foreseeable benefit, retain enough context to avoid changing the observation's meaning, and require each later use to fit the current task. Make correction change behavior and make deletion promises match the information paths that actually exist. The successful system is not the one that mentions the most history. It is the one that uses the right history when it helps—and leaves it alone when it does not.

Limitations

The proposed use contract and evaluation design are engineering recommendations, not a universally validated admission algorithm. The interview evidence reflects participants' recollections rather than a controlled measurement of long-term product benefit.[1] LongMemEval evaluates memory abilities through questions over interaction histories.[2] BenchPreS evaluates appropriate application or suppression of preferences across communication contexts.[4] I recommend separately validating how much users value a particular product's continuity. Supersede's bounded-memory comparison does not establish the behavior of every memory architecture.[5] The Memory FAQ describes a particular product, not a universal deletion guarantee.[6] The evidence supports testing scope, updating, and appropriate use, but does not supply a product-independent retention threshold, expiry schedule, or quantified long-term benefit for this policy.