Chameleon: Geometry-Grounded Episodic Memory for Long-Horizon Robot Manipulation

Paper: 2603.24576 Authors: Xinying Guo, Chenxi Jiang, Hyun Bin Kim, Ying Sun, Yang Xiao, Yuhang Han, Jianfei Yang Categories: cs.RO, cs.AI, cs.CV

The Gap

Current robot manipulation systems hit a wall when the same visual scene can mean different things depending on what happened before. Imagine a robot seeing a closed drawer — did it just close it, or was it already closed? This is perceptual aliasing, and it breaks the Markov assumption that “current observation is enough to decide what to do next.”

Existing memory solutions (like RT-2, ACT, Octo) compress history into semantic summaries or retrieve similar past episodes. The problem: semantic compression throws away the exact visual details that disambiguate “closed drawer after I closed it” from “closed drawer I haven’t touched.” Similarity-based retrieval returns perceptually similar but contextually irrelevant memories — like recalling “any time I saw a closed drawer” instead of “the specific moment I closed this drawer.”

Problem: Perceptual Aliasing
    |
    v
Assumption: Fine-grained geometry + goal-directed recall > semantic compression
    |
    v
Method: Geometry-grounded tokens + differentiable memory stack
    |
    v
Evidence: Camo-Dataset (UR5e robot) shows improved success rates
    |
    v
Conclusion: Preserving perceptual detail + task-aware retrieval solves aliasing

The Increment

One sentence: Before Chameleon, robots either forgot disambiguating visual details or retrieved irrelevant memories; after Chameleon, robots preserve geometry-grounded context and recall goal-directed episodes.

Core Mechanism

Chameleon has three components working in sequence. First, a multimodal encoder processes RGB-D observations and language goals into tokens that preserve geometric structure (point clouds, spatial relationships) rather than collapsing them into abstract features. These tokens get written to a differentiable memory stack — think of it as a tape recorder that keeps the full sensory trace, not just a summary.

Second, when the robot needs to decide what to do, a cross-attention mechanism queries this memory stack. The query is conditioned on both the current observation and the task goal, so retrieval is goal-directed: “find memories relevant to closing this specific drawer” rather than “find any drawer memories.” The attention weights are learned end-to-end, so the system discovers which past moments matter for the current decision.

Observation (RGB-D) + Goal (language)
    |
    v
[Multimodal Encoder]
    |
    v
Geometry-grounded tokens (preserve spatial structure)
    |
    v
[Memory Stack] <--- writes full trace
    |
    v
[Cross-Attention Query] <--- conditioned on current obs + goal
    |
    v
Retrieved context
    |
    v
[Policy Network] ---> Action

Think of Chameleon like a detective’s evidence board. Most systems take crime scene photos and write brief notes (“closed drawer, 3pm”) — that’s semantic compression. When they need to solve a case, they search their notes for “closed drawer” and get every unrelated closed-drawer case — that’s similarity retrieval. Chameleon instead pins the actual photos to the board (geometry-grounded tokens) and uses red string to connect evidence relevant to the current case (goal-directed cross-attention). The detective doesn’t search for “any closed drawer” — they trace connections from “this specific drawer, this specific timeline, this specific goal.”

Key Concepts

• Perceptual Aliasing: When different world states produce identical observations. A robot sees a cup on a table. Is it the cup it just placed there, or a different cup that was already there? The observation is the same, but the correct action differs: if it just placed the cup, it should move to the next task; if the cup was already there, it might need to move it first. Without memory of what happened before, the robot can’t tell these apart. This breaks the Markov property — you can’t choose the right action from observation alone; you need history.

• Geometry-Grounded Tokens: Instead of encoding “there’s a drawer” as an abstract feature vector, preserve the actual 3D structure: where the drawer is in space, its orientation, depth from the camera. This is like keeping the blueprint instead of just writing “rectangular object.” When the robot later needs to distinguish “the drawer I closed” from “a different drawer,” the preserved geometry provides the disambiguating cue — exact position, angle, how it looked when I last interacted with it.

• Goal-Directed Recall: Memory retrieval conditioned on what you’re trying to achieve. If the task is “close the drawer,” the system queries memory for moments relevant to drawer-closing, not just any moment with a drawer. This is implemented via cross-attention where the query vector encodes both current observation and task goal. The attention mechanism learns to weight memories by relevance to the current objective, not just perceptual similarity.

Framework Shift

Before (mainstream approach):        After (Chameleon):

Observation                          Observation + Goal
    |                                    |
    v                                    v
Semantic Encoder                     Geometry Encoder
    |                                    |
    v                                    v
Compressed Summary                   Full Spatial Trace
    |                                    |
    v                                    v
Similarity Search                    Goal-Directed Query
    |                                    v
    v                                Cross-Attention
Any similar memory                       |
    |                                    v
    v                                Relevant context
Policy                               Policy

From lossy compression + blind retrieval to lossless preservation + task-aware recall, the core shift is treating memory as a queryable spatial database rather than a semantic index.

Expert Assessment

Problem choice: Real gap. Perceptual aliasing is a known failure mode in long-horizon manipulation, and existing memory solutions genuinely struggle with it. The problem sits at the intersection of embodied AI and memory-augmented learning — timely given the push toward longer-horizon robot tasks.

Method maturity: The geometry-grounding idea is sound, but the execution feels incremental. Differentiable memory stacks aren’t new (Neural Turing Machines, 2014), and cross-attention for retrieval is standard. The novelty is in the combination and application domain, not the components. A simpler baseline might be: just keep more history in the context window and let a transformer handle it. The paper doesn’t convincingly argue why the stack + cross-attention architecture is necessary over, say, a longer-context policy.

Experimental integrity: Camo-Dataset is a strength — real UR5e robot data with designed perceptual aliasing scenarios. But the baselines are weak: RT-2 and ACT weren’t designed for this problem, so beating them doesn’t prove much. Missing: ablations on what matters (geometry vs. multimodal, stack vs. simple buffer, goal-conditioning vs. not). The success rate improvements are consistent but modest (10-15% in most tasks), suggesting the method helps but isn’t a silver bullet.

Writing quality: The human episodic memory framing in the intro is oversold — the method doesn’t really model episodic memory’s retrieval dynamics. Section 3.2 (method) is dense and would benefit from a worked example. The related work section lumps together disparate approaches without clearly positioning where Chameleon sits.

Verdict: weak accept — Addresses a real problem with a reasonable solution and provides a useful dataset, but the method is more engineering than insight, and the evaluation could be sharper.

Takeaways

The geometry-grounding principle transfers: when building memory systems for embodied agents, preserve spatial structure instead of collapsing to semantic features. This applies beyond robotics — any domain where “what happened where” matters (video understanding, navigation, surgical planning).

The goal-conditioned retrieval pattern is reusable: instead of retrieving by similarity alone, condition your query on the current objective. Implement this via cross-attention where the query encodes both observation and goal. Simple but effective.

The Camo-Dataset design is instructive: if you want to test memory, explicitly create scenarios where the same observation arises from different histories. This “adversarial dataset design” approach — engineer failure modes into your benchmark — is underused in robotics.