Remember to be Curious: Persistent 3D Memory Solves Exploration Amnesia

Paper: 2605.22814 Authors: Lily Goli, Justin Kerr, Daniele Reda, Alec Jacobson, Andrea Tagliasacchi, Angjoo Kanazawa Categories: cs.LG

The Gap

Curiosity-driven RL works beautifully in simple gridworlds: agents get intrinsic rewards for visiting novel states, gradually covering the space. But scale this to photorealistic 3D environments and it collapses. The agent spins in circles, gets “surprised” by the same wall from a slightly different angle, and never systematically explores.

Prior work (ICM, RND, NGU) uses prediction error or state novelty as curiosity signals. The problem: these methods treat each observation independently, with no persistent spatial memory. The agent forgets where it’s been after a few episodes. It also lacks trajectory context—no sense of “I just came from the kitchen, so I should head toward unexplored rooms, not back to the kitchen.”

The gap: curiosity needs memory. Both spatial memory (what parts of the world exist) and episodic memory (where I’ve been this episode).

Problem: Curiosity-driven exploration fails in 3D photorealistic worlds
   |
   v
Diagnosis: Agents lack (1) persistent spatial memory
                        (2) episodic trajectory context
   |
   v
Method: (1) Online 3D reconstruction as world model
        (2) Sequence model over RGB frames as policy
   |
   v
Evidence: Outperforms active mapping baselines on HM3D
          Zero-shot transfer to Gibson + AI-generated worlds
   |
   v
Conclusion: Persistent + episodic memory enables effective curiosity

The Increment

One sentence: Before this paper, curiosity-driven agents explored 3D worlds with amnesia; after, they maintain a persistent 3D map and episodic trajectory memory, enabling systematic exploration and zero-shot generalization.

Core Mechanism

The system has two components that run in parallel during training:

Persistent world model: An online 3D Gaussian Splatting reconstruction that accumulates observations across all episodes. As the agent explores, new RGB-D frames get fused into this 3D model. The model renders novel views, and the mismatch between rendered and actual observations generates curiosity rewards. Crucially, this model persists—it doesn’t reset between episodes, so the agent can’t get “surprised” by previously seen geometry.

Episodic policy: A transformer-based policy that takes a sequence of RGB frames (the agent’s trajectory so far this episode) and outputs actions. No explicit map input at test time—just raw pixels. The sequence modeling gives the agent implicit memory of where it’s been this episode, steering it away from immediate backtracking.

Training loop:
                                    
  RGB-D frames -----> 3D Gaussian Splatting (persistent)
       |                        |
       |                        v
       |                  Render novel view
       |                        |
       |                        v
       +---> Compare -----> Prediction error = Curiosity reward
       |                        |
       v                        v
  Transformer policy <---- RL update (PPO)
  (sequence of RGB)
       |
       v
  Action --> Environment

At deployment, the 3D model is discarded. The policy runs on RGB frames alone, using its learned exploration strategy.

Structural metaphor: Think of a museum visitor with a notebook and a growing mental map. The 3D reconstruction is the mental map—it accumulates over multiple visits (episodes), so you remember which wings you’ve already seen. The transformer policy is the notebook where you jot down your path today—“entered through the main hall, went left to impressionism, now heading right.” The notebook resets each visit, but the mental map persists. When you see a hallway, you check your mental map (render what you expect to see). If reality doesn’t match, that’s interesting—go explore. The notebook keeps you from immediately doubling back. After many visits, you’ve internalized efficient exploration patterns, so you can navigate a new museum without the mental map, just using your notebook and instincts.

Key Concepts

• Curiosity as prediction error: Imagine you’re walking through a house blindfolded, building a mental model as you go. You predict what the next room looks like based on what you’ve seen. When you remove the blindfold and reality doesn’t match your prediction, that’s surprising—that’s curiosity. In RL, this surprise becomes an intrinsic reward. The agent is motivated to visit places where its world model is wrong, which naturally drives exploration. The catch: if your mental model resets every day, you’ll be “surprised” by the same kitchen over and over. That’s why persistence matters.

• Episodic vs semantic memory: Episodic memory is “I walked through the kitchen 30 seconds ago.” Semantic memory is “kitchens have stoves and fridges.” In this paper, the 3D reconstruction is semantic (persistent facts about the world’s geometry), while the transformer’s sequence context is episodic (where I’ve been this episode). The episodic memory prevents local loops (“I just came from there, don’t go back”), while the semantic memory prevents global loops (“I’ve fully explored that wing across all episodes”).

• 3D Gaussian Splatting: A way to represent 3D scenes as a collection of oriented, colored “splats” (think fuzzy ellipsoids). Unlike neural radiance fields (NeRF), which are slow to render, Gaussian Splatting is fast and can be updated online as new observations arrive. Each splat has a position, color, opacity, and covariance. Rendering a view is just projecting these splats onto the image plane and blending them. For this paper, it’s the data structure that holds the persistent world model.

Framework Shift

Before (standard curiosity):        After (this paper):

Episode 1:                          Episode 1:
  Observe --> Predict --> Error       Observe --> 3D Model (empty)
      |           |                       |            |
      v           v                       v            v
  Reward <-- Forget                   Render --> Error --> Reward
                                          |
Episode 2:                                v
  Observe --> Predict --> Error       3D Model (updated)
      |           |                       |
      v           v                   Episode 2:
  Reward <-- Forget (same error!)       Observe --> 3D Model (persists)
                                            |            |
[No memory across episodes]                 v            v
[No trajectory context]                 Render --> Error --> Reward
                                            |            (lower for seen areas)
                                            v
                                        3D Model (grows)
                                        
                                        + Transformer sees trajectory
                                          (episodic context)

From stateless curiosity to memory-augmented curiosity, the core shift is treating exploration as a cumulative process rather than a per-episode reset.

Expert Assessment

Problem choice: This is a real gap. Curiosity-driven RL has struggled in complex 3D environments for years, and the “exploration amnesia” diagnosis is spot-on. The problem sits at the intersection of RL exploration and 3D vision, which is timely given the push toward embodied AI. Not a manufactured gap.

Method maturity: The combination is clever but not deeply novel—3D reconstruction for world models has been explored (e.g., in SLAM-based navigation), and sequence models for policies are standard. The insight is in the pairing: persistent reconstruction + episodic policy. The execution is solid, but I’d want to see ablations on simpler alternatives (e.g., voxel grids instead of Gaussian Splatting, or explicit episodic maps). The paper doesn’t deeply explore failure modes—what happens when the 3D model drifts or when the environment is highly dynamic?

Experimental integrity: Baselines are reasonable (active mapping methods like ANS, which also use 3D reconstruction but with explicit planning). The zero-shot transfer to Gibson and AI-generated worlds is impressive and suggests the method isn’t overfitting to HM3D. However, the downstream task results (apple picking, image-goal navigation) show modest improvements over from-scratch baselines—this suggests the learned exploration policy transfers, but not dramatically. The paper could be more transparent about compute costs (training a 3D reconstruction online is expensive).

Writing quality: The paper is well-structured, but the related work section is dense and could be trimmed. The method section would benefit from a clearer breakdown of what happens at training vs deployment—it’s easy to miss that the 3D model is discarded at test time. The ablation study is thorough, but the figures are small and hard to parse. Rewriting the results section to lead with the zero-shot transfer (the most striking result) would strengthen the narrative.

Verdict: weak accept — Solid execution on a real problem with convincing zero-shot generalization, but the method is more “good engineering” than conceptual breakthrough. The downstream task results are underwhelming, and the paper doesn’t deeply probe when this approach fails.

Takeaways

Persistent world models for exploration: If you’re training agents in 3D environments, don’t reset your world model between episodes. Accumulate spatial knowledge across episodes to avoid redundant exploration. This applies beyond RL—any system that needs to explore a space (active learning, data collection) benefits from persistent memory.

Sequence models for implicit episodic memory: Instead of explicitly tracking visited states, use a transformer over recent observations. It’s simpler than maintaining an explicit episodic map and naturally handles partial observability. This is a lightweight way to add “where I’ve been recently” context to any policy.

Curiosity needs both semantic and episodic memory: Semantic memory (persistent facts) prevents global redundancy. Episodic memory (recent trajectory) prevents local loops. You need both. This framing transfers to other exploration problems—e.g., hyperparameter search, where you want to remember what you’ve tried globally but also avoid revisiting nearby configurations.

Zero-shot transfer as a test of learned exploration: The fact that the policy generalizes to unseen environments without the 3D model suggests it learned general exploration heuristics (e.g., “move toward open space,” “avoid backtracking”). This is a useful lens for evaluating whether your exploration method learned transferable strategies vs environment-specific tricks.