Chameleon: Episodic Memory for Long-Horizon Robotic Manipulation

Paper: 2603.24576 Authors: Xinying Guo, Chenxi Jiang, Hyun Bin Kim, Ying Sun, Yang Xiao, Yuhang Han, Jianfei Yang Categories: cs.RO, cs.AI, cs.CV

Abstract

Robotic manipulation often requires memory: occlusion and state changes can make decision-time observations perceptually aliased, making action selection non-Markovian at the observation level because the same frame may require different actions depending on history. This paper proposes Chameleon, a memory system that stores geometry-aware multimodal tokens and uses a differentiable memory stack to retrieve relevant episodes for the current goal. It also introduces Camo-Dataset, a real UR5e robot dataset for episodic recall, spatial tracking, and sequential manipulation under perceptual ambiguity. The reported results indicate improved reliability and stronger long-horizon control versus baselines.

Key Contributions

• Frames long-horizon manipulation as a memory retrieval problem under perceptual ambiguity
• Introduces geometry-grounded multimodal memory tokens for preserving disambiguating context
• Uses a differentiable memory stack for goal-directed episodic recall
• Releases Camo-Dataset for episodic recall and sequential manipulation evaluation
• Reports improved long-horizon control over baseline methods

Methodology

The main idea is that a robot cannot always act correctly from the current observation alone. If two scenes look the same but require different actions, the missing ingredient is history. Chameleon addresses this by storing structured memories that preserve both visual and geometric information, then retrieving the episode that best matches the current goal.

The differentiable memory stack is important because it lets the system use memory without turning recall into a brittle hard lookup. In effect, the robot can keep a compact history of useful episodes and revisit them when the scene becomes ambiguous.

Results

The paper evaluates the system on Camo-Dataset, a real UR5e dataset designed to stress episodic recall and spatial tracking. The results show that Chameleon improves reliability and control in settings where perception alone is insufficient.

The broader takeaway is that memory is not just for language agents. For manipulation, episodic memory can be the difference between a task that looks Markovian on paper and one that is actually solvable in the real world.

Takeaways

1. Long-horizon manipulation often needs memory because observations alone can be ambiguous
2. Geometry-aware episodic memory helps preserve the context needed for correct action selection
3. Differentiable memory stacks make retrieval trainable rather than brittle
4. Real robot datasets are essential for evaluating memory under perceptual confusion
5. Episodic recall can materially improve long-horizon robotic control