ParamMem: Augmenting Language Agents with Parametric Reflective Memory

Paper: 2602.23320 Authors: Tianjun Yao, Yongqiang Chen, Yujia Zheng, Pan Li, Zhiqiang Shen, Kun Zhang Categories: cs.LG, cs.MA

Abstract

Self-reflection has emerged as a critical capability for language agents, enabling iterative solution refinement. However, existing reflection mechanisms often fall into repetitive patterns that constrain reasoning performance. This paper introduces ParamMem, a parametric memory module that addresses the diversity bottleneck in agent reflection by encoding cross-sample reflection patterns directly into model parameters. Through temperature-controlled sampling, ParamMem generates diverse reflection signals that break the cycle of repetitive outputs. The authors propose ParamAgent, a comprehensive framework integrating parametric memory with episodic and cross-sample memory systems. Empirical validation across code generation, mathematical reasoning, and multi-hop question answering tasks demonstrates consistent performance gains over state-of-the-art baselines, with notable sample efficiency and weak-to-strong transfer capabilities.

Key Contributions

• Parametric Memory Architecture: A novel memory module that encodes reflection patterns into model parameters, enabling diverse reflection generation through stochastic sampling mechanisms
• Empirical Correlation Analysis: Systematic demonstration of the positive relationship between reflective diversity and task success rates across multiple reasoning domains
• ParamAgent Framework: An integrated agent architecture combining parametric, episodic, and cross-sample memory for enhanced reflection capabilities
• Self-Improvement Without External Models: A training paradigm that achieves performance gains without reliance on stronger external models for supervision
• Cross-Scale Transfer: Evidence of weak-to-strong generalization, where smaller models can effectively transfer learned reflection patterns to larger architectures

Methodology and Architecture

The ParamMem module operates by fine-tuning a language model on a curated dataset of successful reflection trajectories. Unlike traditional episodic memory that retrieves past experiences, parametric memory internalizes reflection patterns within the model’s weights. This approach offers several architectural advantages:

The training process involves collecting reflection-action pairs from successful task completions, then fine-tuning the base model to predict effective reflections given task context. During inference, temperature-controlled sampling from this parametric distribution generates diverse reflection candidates. The diversity mechanism is crucial—higher temperatures produce more varied reflections, preventing the agent from cycling through identical reasoning paths.

ParamAgent extends this foundation by maintaining three complementary memory systems: (1) parametric memory for pattern-based reflection generation, (2) episodic memory for storing task-specific experiences, and (3) cross-sample memory for aggregating insights across multiple problem instances. This multi-tier architecture balances generalization with specificity, allowing agents to leverage both learned patterns and concrete examples.

Experimental Results and Analysis

The evaluation spans three challenging domains: code generation (HumanEval, MBPP), mathematical reasoning (GSM8K, MATH), and multi-hop question answering (HotpotQA). Across all benchmarks, ParamAgent demonstrates consistent improvements over baseline reflection methods, with particularly strong gains in mathematical reasoning tasks where diverse solution strategies prove most valuable.

A key finding is the sample efficiency of ParamMem. While traditional reflection methods require extensive trial-and-error during inference, parametric memory amortizes this exploration cost during training. The authors show that ParamAgent achieves comparable or superior performance with significantly fewer inference-time reflection iterations.

The weak-to-strong transfer experiments reveal an intriguing property: reflection patterns learned by smaller models (e.g., 7B parameters) successfully transfer to larger models (e.g., 13B, 70B parameters) with minimal performance degradation. This suggests that effective reflection strategies exhibit scale-invariant characteristics, opening possibilities for efficient knowledge distillation in agent systems.

Ablation studies isolate the contribution of each memory component. Parametric memory alone provides substantial gains, while the combination with episodic and cross-sample memory yields the best overall performance. The temperature parameter emerges as a critical hyperparameter—moderate temperatures (0.7-0.9) balance diversity with coherence, while extreme values either collapse to deterministic outputs or produce incoherent reflections.

Implications for Agent Design

ParamMem represents a shift in how we conceptualize agent memory systems. Traditional approaches treat memory as external storage—databases of past experiences to be retrieved and reused. Parametric memory instead views the model itself as a malleable substrate for encoding strategic knowledge. This perspective aligns with recent work on in-context learning and prompt engineering, but operates at a deeper architectural level.

The self-improvement capability without external supervision is particularly significant for practical deployment. Many reflection-based agents rely on stronger models (e.g., GPT-4) to provide feedback or generate training data. ParamMem’s ability to bootstrap from its own successful trajectories suggests a path toward more autonomous agent development, reducing dependence on proprietary systems.

The diversity-performance correlation documented in this work has broader implications for agent design. It suggests that exploration mechanisms—traditionally associated with reinforcement learning—play an equally important role in language-based reasoning. Future agent architectures might benefit from explicit diversity objectives during both training and inference.

Takeaways

1. Reflective diversity strongly correlates with task success in language agents, motivating architectural designs that explicitly promote varied reasoning paths
2. Parametric memory offers a sample-efficient alternative to episodic retrieval by encoding reflection patterns directly into model weights
3. Temperature-controlled sampling from parametric distributions provides a simple yet effective mechanism for generating diverse reflections
4. Multi-tier memory architectures combining parametric, episodic, and cross-sample components achieve superior performance compared to single-memory systems
5. Weak-to-strong transfer of reflection patterns suggests scale-invariant properties of effective reasoning strategies, enabling efficient knowledge distillation
6. Self-improvement without external model supervision is achievable through careful curation of successful reflection trajectories
7. The ParamAgent framework demonstrates consistent gains across diverse reasoning domains, indicating broad applicability of the parametric memory approach