MM-Lifelong: Teaching AI to Remember Across Days, Weeks, and Months

Paper: 2603.05484 Authors: Guo Chen, Lidong Lu, Yicheng Liu, Liangrui Dong, Lidong Zou, Jixin Lv, Zhenquan Li, Xinyi Mao, Baoqi Pei, Shihao Wang Categories: cs.CV

The Gap

Video understanding datasets have grown to hours of footage, but they’re stitched-together highlight reels—dense, scripted clips that don’t reflect how life actually unfolds. Real days have dead time: you cook breakfast, commute, work, sleep. Events are sparse, separated by hours or days. Current models trained on MovieQA or EgoSchema never learn to handle this sparsity.

Existing approaches fail in two ways. End-to-end multimodal LLMs (like GPT-4V or Gemini) try to cram everything into context, hitting a working memory bottleneck around 10-20 minutes of video. Agentic systems (like Reflexion or Voyager) navigate by jumping through timestamps, but when events are a week apart, they lose their bearings—what the authors call “global localization collapse.” You can’t binary-search your way through a month when you don’t know where anything is.

Problem: Dense datasets != sparse life
   |
   v
Assumption: Real understanding needs multi-timescale memory
   |
   +---> Method: ReMA (recursive belief state + dynamic memory)
   |
   v
Evidence: Outperforms MLLMs and agents on Day/Week/Month tasks
   |
   v
Conclusion: Hierarchical memory > flat context or blind search

The Increment

One sentence: Before, models either drowned in context or got lost in time; now, we have a dataset that exposes both failure modes and an agent that maintains a recursive belief state to navigate sparse, month-long timelines.

Core Mechanism

ReMA operates like a detective building a case file. It doesn’t try to hold everything in working memory or blindly search through evidence. Instead, it maintains a recursive belief state—a compressed summary of what it knows so far—and updates it as new information comes in.

The system has three components: a perception module that extracts visual and audio features from video segments, a memory manager that decides what to keep and what to discard, and a reasoning engine that queries the belief state to answer questions. When you ask “What did I eat for breakfast three days ago?”, ReMA doesn’t replay three days of footage. It checks its belief state for breakfast-related entries, retrieves the relevant compressed memory, and answers.

The key innovation is dynamic memory management. As the timeline grows, ReMA doesn’t just append new observations—it recursively merges and abstracts them. Early morning routines get compressed into “typical weekday morning.” Anomalies (like a doctor’s appointment) stay detailed. This hierarchical compression prevents both context overflow and localization collapse.

Query: "What happened Tuesday morning?"
   |
   v
[Belief State] <--- recursive summary of past observations
   |            |
   v            +---> [Memory Manager]
[Reasoning]              |
   |                     v
   v              [Perception Module]
Answer                   |
                         v
                   Video segments

Think of ReMA as a journalist covering a month-long story. A bad journalist either takes verbatim notes on everything (context overflow) or skims randomly hoping to find the scoop (localization collapse). A good journalist maintains a running narrative—a belief state—that gets updated with each interview. Early background details get summarized; breaking news stays granular. When the editor asks for a specific fact, the journalist doesn’t re-read all notes; they consult their structured narrative and drill down where needed. ReMA’s belief state is that narrative, and its memory manager is the editorial judgment deciding what to compress and what to preserve.

Key Concepts

• Working Memory Bottleneck: Imagine trying to remember a phone conversation word-for-word while having three more conversations. At some point, you can’t hold it all. MLLMs face this with long videos—they have a fixed context window (say, 128K tokens), and once video frames fill it up, they start forgetting earlier content or refusing to process more. It’s not a training problem; it’s an architectural ceiling. The model literally runs out of space to “think.”

• Global Localization Collapse: You’re looking for a specific scene in a month of footage. You jump to day 15—too late. Jump to day 7—too early. Jump to day 11—still wrong. Without landmarks, you’re guessing. Agentic systems that navigate by timestamp jumps face this: when events are sparse (one relevant moment per day), random search becomes exponentially expensive. They collapse because they can’t build a mental map of where things are in time.

• Recursive Belief State: Instead of storing raw observations, you store a summary that gets refined. Day 1: “Woke up, made coffee, worked.” Day 2: “Woke up, made coffee, worked.” After a week, this compresses to “Typical work routine: wake, coffee, work.” But if Day 5 had a dentist appointment, that stays detailed. The belief state is recursive because each level summarizes the level below—hours compress into days, days into weeks. It’s lossy, but intelligently so.

Framework Shift

Before (mainstream approach):        After (this paper):

[MLLM: flat context window]         [ReMA: hierarchical belief]
                                     
Video ---> [==========] ---> Answer  Video ---> [Perception]
           (context fills up)                        |
                                                     v
                                              [Memory Manager]
                                                     |
                                                     v
                                              [Belief State]
                                               /    |    \
                                            Day  Week  Month
                                                     |
                                                     v
                                              [Reasoning] ---> Answer

OR

[Agent: blind search]
Video timeline: [----*-------*---------*----]
                     ^       ^         ^
                  (jump)  (jump)    (jump)
                  (lost)  (lost)    (found?)

From treating video as a flat stream to be consumed or searched, to building a hierarchical memory structure that mirrors how humans actually remember life—compressed routines with salient details preserved.

Expert Assessment

Problem choice: This is a real gap. The field has been stuck in “movie understanding” mode—dense, scripted content—while ignoring that most video data (surveillance, lifelogs, robotics) is sparse and unstructured. The authors correctly identify that scaling context windows won’t solve this; you need architectural changes. It’s timely given the push toward embodied AI and personal assistants.

Method maturity: ReMA is more clever than brute force, but it’s not radically novel—recursive summarization has been explored in NLP (hierarchical attention, memory networks). The contribution is adapting it to multimodal, multi-timescale video. I’d like to see ablations on the memory manager’s compression strategy—how much is learned vs. hand-tuned? The paper hints at “dynamic” management but doesn’t fully open the hood.

Experimental integrity: The baselines are fair—they test both end-to-end MLLMs (GPT-4V, Gemini) and agentic systems (Reflexion-style). The failure mode analysis (working memory bottleneck, localization collapse) is well-documented with error breakdowns. One concern: 181 hours is decent but not massive. Does ReMA’s advantage hold at 1000+ hours, or does it hit its own scaling wall? The dataset splits for temporal/domain bias are a nice touch, though out-of-distribution results aren’t deeply explored.

Writing quality: The abstract and intro are crisp. The method section gets a bit hand-wavy around the memory manager—phrases like “intelligently compresses” without pseudocode or loss functions. The related work could be tighter; it lists papers without clearly positioning what’s different. If I were reviewing, I’d ask for a detailed algorithm box for ReMA and more ablation studies on compression strategies.

Verdict: weak accept — Solid dataset contribution and a reasonable baseline that exposes real failure modes, but the method feels like an incremental step rather than a breakthrough; needs deeper technical exposition.

Takeaways

If you’re building any system that processes long-duration, sparse data—think security footage analysis, personal health monitoring, or long-context document understanding—steal the hierarchical belief state idea. Don’t try to fit everything into a flat context or search blindly. Instead, maintain a compressed summary at multiple timescales and update it incrementally. The specific trick: make compression adaptive, not uniform. Keep anomalies detailed, compress routines aggressively.

Also, the dataset design is instructive: structuring data across Day/Week/Month scales forces models to handle varying temporal densities. If you’re creating benchmarks, consider multi-timescale splits—they reveal failure modes that single-scale datasets hide.