Does Your Agent's Memory Survive a Model Upgrade?

Paper: 2609.05339 Authors: Ankit Goyal, Jaideep Ray Categories: cs.AI, cs.CL, cs.IR

The Gap

Upgrading the underlying large language model of an autonomous agent is routine engineering: switching from an older release to a newer, smarter model promises immediate performance gains. But what happens to the agent’s persistent memory bank accumulated over months of user interactions?

Industry practice treats the memory store as a decoupled database: developers assume that as long as memory text or vector embeddings are preserved, a smarter model will read them better.

This paper proves that assumption false. An agent can retain the exact same memory store and still suffer catastrophic functional amnesia. When a new model acts as the reader of notes written by an older model, semantic idiosyncrasies cause severe interpretation drift. Furthermore, in retrieval-augmented memory (RAG), mixing embeddings across model revisions breaks similarity geometry, while compressed natural-language notes (NOTES) permanently drop crucial raw context that cannot be repaired after the fact without the original interaction logs.

[AGENT LIFECYCLE] Interactions over Months -> Persistent Memory Bank
                       |
                       v
         [UPGRADE BASE MODEL: M_old -> M_new]
                       |
    +------------------+------------------+
    v                                     v
[COMMON ASSUMPTION]                 [OBSERVED REALITY]
"Memory is text; a smarter          - Reader/Writer interpretation drift (-13.3 pts)
 model will read old notes          - Mixed-embedding vector distortion in RAG
 even better than the old one."      - 80% of note failures due to unrecoverable
                                      initial compression loss!

The Increment

One sentence: Before this paper, agent memory was presumed portable across model swaps; after it, LinkedIn’s controlled audit demonstrates that free-form natural language notes suffer severe, asymmetric degradation under model upgrades, establishing that only fixed-schema knowledge graphs (KG-fixed) guarantee true zero-drift memory portability.

Core Mechanism

The authors designed a rigorous benchmark evaluating 48 synthetic interaction histories with randomized answer codes and exact match scoring across sub-10B open-weight models, comparing four fundamental memory representations:

1. LC-RAW: Preserving the verbatim interaction history in full-context window.
2. RAG: Chunked semantic retrieval over past turns.
3. NOTES: Free-form natural language summaries synthesized by the model after each session.
4. KG-fixed: Facts parsed into strict, typed relational triplets (Subject-Predicate-Object).

The experimental findings reveal critical structural failure modes:

• Asymmetric Writer-Reader Coupling: When swapping models, NOTES performance swung wildly by +9.91% in one direction but plunged by -13.28% in reverse. A newer model often fails to parse the implicit shortcuts and syntactic conventions of an older model.
• Root Cause of Deficit: Diagnostic decomposition proved that 80% (0.467 ± 0.014) of the accuracy deficit in compressed notes stems from information permanently discarded during initial note-taking, not downstream reading errors.
• The Mixed-Index Trap: In RAG stores, a lazy migration using a 50/50 mix of old and new vector embeddings captured only 4.96 points of improvement, forfeiting over half of the 11.90-point gain achieved by fully re-indexing the dataset.
• Store-Only Repair Failure: Attempting to “repair” or rewrite corrupted notes using only the note store failed in 100% (0/48) of test cases. Repair only succeeded (34/48 cases) when the agent had retained the raw, uncompressed source history.

  MEMORY PORTABILITY UNDER MODEL MIGRATION (M_1 -> M_2)

  Memory Type       Portability Delta    Root Bottleneck
  +---------------+--------------------+-----------------------------+
  | KG-fixed      | +0.0004 +/- 0.0020 | None (Deterministic schema) |
  | LC-RAW        | Stable             | Context window cost         |
  | RAG           | -5.94 to +11.90    | Embedding space fracture    |
  | NOTES         | -13.28 to +9.91    | Lossy initial compression   |
  +---------------+--------------------+-----------------------------+

To explain this dynamic, consider a structural metaphor of medical patient charts. A physician who scribbles informal, personalized shorthand into a paper notebook (the NOTES approach) understands their own abbreviations perfectly. When a new doctor takes over the practice, they misinterpret the shorthand dosage instructions and prescribe the wrong medicine. If the clinic had instead recorded vital signs into a standardized digital schema with strict blood pressure and allergy fields (KG-fixed), any doctor in the world could step in with zero confusion.

Key Concepts

• Memory Portability: The degree to which an agent’s historical memory store preserves its retrieval and task completion fidelity when the underlying LLM backbone is replaced.
• Asymmetric Interpretation Drift: The empirical reality that Model B reading Model A’s summaries does not yield the same accuracy as Model A reading Model B’s summaries.
• Lossy Compression Debt: The irreversible loss of nuance during initial natural-language summarization that makes future memory reconstruction impossible without raw source history.

Framework Shift

Before (Naive Memory Persistence):
Raw History -> [Model A Summarizes] -> [Text Notes] -> [Model B Reads] -> Failure!
(Assumes natural language is a universal, model-agnostic storage protocol)

After (Architected Memory Portability):
Raw History -> [Schema Extraction] -> [Fixed-Schema KG] -> [Any Model Reads] -> Zero Drift!
     |
     +-------> [Raw Log Cold Storage Archive] (Enables full replay & future re-indexing)

From treating free-form text summaries as a universal agent memory format to recognizing the necessity of typed schemas and raw-log retention, the core shift is treating agent memory as an enterprise database migration problem.

Expert Assessment

Problem choice: Vital and timely. Thousands of production agent architectures (MemGPT, babyAGI derivatives) rely on recursive natural language summarization without realizing they are painting themselves into a vendor lock-in corner.

Method maturity: The experimental design is exemplary. Using 48 controlled synthetic histories with randomized ground truth isolates memory transfer from model knowledge hallucination.

Experimental integrity: The distinction between raw history retention, chunked RAG, and compressed notes provides a crystal-clear causal breakdown of where information actually disappears.

Writing quality: Clear, practical, and highly prescriptive. The paper offers actionable guidance for systems engineers.

Verdict: strong accept — A foundational paper for the long-term systems engineering of stateful AI agents.

Takeaways

• Never discard raw interaction logs. Free-form summaries cannot be repaired once the base model is retired.
• For high-stakes long-term agent memory, normalize entities into a fixed-schema knowledge graph rather than free-form prose.
• When upgrading embedding models in RAG memory, re-index 100% of vectors; mixed-version vector indexes break distance metrics and ruin retrieval precision.