MemCollab: Cross-Agent Memory Collaboration via Contrastive Trajectory Distillation

Paper: 2603.23234 Authors: Yurui Chang, Yiran Wu, Qingyun Wu, Lu Lin Categories: cs.AI

Abstract

LLM-based agents rely on memory mechanisms to reuse knowledge from past problem-solving experiences. Existing approaches construct memory per-agent, tightly coupling stored knowledge to a single model’s reasoning style. In modern deployments with heterogeneous agents, a natural question arises: can a single memory system be shared across different models? Naive memory transfer degrades performance because memory entangles task-relevant knowledge with agent-specific biases. MemCollab addresses this by constructing agent-agnostic memory through contrasting reasoning trajectories from different agents on the same task, distilling abstract reasoning constraints that capture shared invariants while suppressing agent-specific artifacts.

Key Contributions

• Agent-agnostic memory: Constructs memory that transfers across heterogeneous models without degradation
• Contrastive trajectory distillation: Compares reasoning traces from different agents on the same task to extract universal invariants
• Task-aware retrieval: Conditions memory access on task category, ensuring only relevant constraints are used
• Cross-model-family transfer: Works even across different model architectures and sizes

The Problem: Memory is Agent-Specific

When you build memory from an agent’s own reasoning traces, that memory encodes:

• The model’s preferred solving strategies
• Habitual heuristics specific to that architecture
• Reasoning patterns that may not generalize

Experiment on MATH500 shows:

• Qwen2.5-7B + its own memory → improvement
• Qwen2.5-7B + memory from Qwen2.5-32B → worse than no memory
• This happens because the 32B model’s reasoning patterns don’t match how the 7B model thinks

MemCollab’s Approach

Contrastive Distillation

Given the same task solved by multiple agents:

1. Collect reasoning trajectories from each agent
2. Identify invariances — patterns that appear across all agents (likely task-level truths)
3. Identify discrepancies — patterns unique to individual agents (likely agent-specific biases)
4. Distill abstract reasoning constraints that capture only the invariances

This produces memory entries that encode “what must be true about solving this type of problem” rather than “how model X prefers to solve problems.”

Task-Aware Retrieval

Memory entries are tagged with task categories. At inference time, retrieval is conditioned on the incoming task’s category, ensuring the agent receives only relevant reasoning constraints — not a random sample of past experience.

Results

Across mathematical reasoning and code generation benchmarks:

• MemCollab memory outperforms memory distilled from a model’s own trajectories
• Improvements hold for both accuracy and inference-time efficiency
• Works in cross-model-family settings (different architectures benefiting from shared memory)
• The collaboratively constructed memory functions as a shared reasoning resource

Takeaways

• Per-agent memory is fundamentally limited — it bakes in model-specific biases that prevent transfer
• Contrastive distillation across agents is key: by comparing how different models solve the same problem, you isolate what’s universally true vs. what’s one model’s quirk
• This enables a production pattern where expensive large-model reasoning experiences can benefit cheap small-model inference, without the degradation of naive transfer
• Task-aware retrieval is essential — not all past experience is relevant, and irrelevant memories hurt