Anatomy of Agentic Memory: Taxonomy and Empirical Analysis of Evaluation and System Limitations

Paper: 2602.19320 Authors: Dongming Jiang, Yi Li, Songtao Wei, Jinxin Yang, Ayushi Kishore, Alysa Zhao, Dingyi Kang, Xu Hu, Feng Chen, Qiannan Li Categories: cs.CL, cs.AI

Abstract

This survey provides a critical examination of agentic memory systems that enable LLM agents to maintain state across extended interactions. While these systems promise to overcome fixed context window limitations and support long-horizon reasoning, their empirical performance often falls short of theoretical expectations. The authors present a structured taxonomy of Memory-Augmented Generation (MAG) systems based on four core memory structures, then systematically analyze the pain points that limit current implementations: underscaled benchmarks that saturate quickly, evaluation metrics misaligned with semantic utility, significant performance variance across different backbone models, and substantial latency and throughput overhead from memory maintenance operations. By connecting architectural choices to empirical limitations, this work clarifies the gap between promise and practice in agentic memory systems.

Key Contributions

• A concise taxonomy of MAG systems organized around four fundamental memory structures, providing a unified framework for understanding diverse architectural approaches
• Systematic identification of benchmark saturation effects, demonstrating how existing evaluation datasets fail to capture the complexity of real-world memory-dependent tasks
• Analysis of metric validity and judge sensitivity issues, revealing how current evaluation approaches may not accurately reflect semantic utility
• Empirical evidence of backbone-dependent accuracy variations, showing that memory system performance is not architecture-agnostic
• Quantification of system-level costs including latency and throughput overhead introduced by memory maintenance operations

Memory Structure Taxonomy

The paper introduces a four-category taxonomy that organizes agentic memory systems by their underlying memory structures. This classification moves beyond surface-level architectural differences to focus on fundamental organizational principles.

The first category encompasses systems using flat memory structures, where information is stored without hierarchical organization. These systems prioritize simplicity and direct access but may struggle with scalability as memory grows. The second category includes hierarchical memory structures that organize information in tree-like or nested arrangements, enabling more efficient retrieval through structured navigation but introducing complexity in maintenance.

The third category covers graph-based memory structures that represent relationships between memory elements explicitly. These systems excel at capturing complex interdependencies but face challenges in efficient traversal and update operations. The fourth category comprises hybrid approaches that combine multiple structural paradigms to balance different performance characteristics.

This taxonomy reveals that architectural choices in memory structure directly influence the types of empirical limitations systems encounter, creating a foundation for understanding why certain designs underperform in specific scenarios.

Empirical Limitations and Pain Points

The survey identifies four critical pain points that explain the performance gap in current agentic memory systems.

Benchmark saturation emerges as a significant issue, where existing evaluation datasets are too small or too simple to meaningfully differentiate between systems. Many benchmarks can be solved with relatively basic memory mechanisms, failing to stress-test the sophisticated capabilities that advanced systems claim to provide. This saturation effect masks real performance differences and provides false confidence in system capabilities.

Metric validity and judge sensitivity present another challenge. Current evaluation metrics often focus on surface-level correctness rather than semantic utility. When using LLM-as-judge approaches, the choice of judge model significantly impacts evaluation outcomes, introducing instability and making cross-study comparisons difficult. Metrics may reward systems that produce plausible-sounding but semantically incorrect responses.

Backbone-dependent accuracy reveals that memory system performance varies dramatically based on the underlying LLM. A memory architecture that performs well with one backbone model may fail with another, suggesting that memory mechanisms are not as modular or transferable as often assumed. This dependency complicates system design and deployment decisions.

System-level costs including latency and throughput overhead are frequently overlooked in evaluations that focus solely on accuracy. Memory maintenance operations—retrieval, update, consolidation—introduce substantial computational costs that can make systems impractical for real-world deployment despite strong accuracy metrics.

Implications for Future Research

This analysis has several important implications for advancing agentic memory systems. First, the field needs more challenging, diverse benchmarks that can differentiate between sophisticated memory mechanisms. These benchmarks should include tasks requiring genuine long-horizon reasoning and complex state maintenance.

Second, evaluation methodologies must evolve beyond simple accuracy metrics to capture semantic utility, system efficiency, and robustness across different backbone models. Multi-dimensional evaluation frameworks that consider accuracy, latency, throughput, and memory efficiency simultaneously would provide more realistic performance pictures.

Third, architectural research should explicitly account for backbone model characteristics when designing memory systems. Rather than pursuing one-size-fits-all solutions, the field may benefit from developing memory architectures optimized for specific model families or capabilities.

Fourth, system-level optimization deserves equal attention to algorithmic innovation. Reducing memory maintenance overhead through efficient indexing, caching strategies, and incremental update mechanisms could unlock practical deployment of theoretically sound approaches.

Takeaways

1. Current agentic memory systems underperform their theoretical promise due to systematic evaluation and implementation limitations rather than fundamental architectural flaws
2. The four-category memory structure taxonomy (flat, hierarchical, graph-based, hybrid) provides a unified framework for understanding diverse MAG system designs
3. Benchmark saturation is a critical issue—existing evaluation datasets are too simple to meaningfully differentiate between sophisticated memory mechanisms
4. Evaluation metrics often misalign with semantic utility, and LLM-as-judge approaches introduce significant sensitivity to judge model selection
5. Memory system performance exhibits strong backbone model dependency, challenging assumptions about architectural modularity and transferability
6. System-level costs including latency and throughput overhead are frequently overlooked but can render otherwise accurate systems impractical for deployment
7. Future progress requires simultaneous advances in benchmark design, evaluation methodology, backbone-aware architecture, and system-level optimization