Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games

Paper: 2606.19338 Authors: Shengyuan Ding, Xilin Wei, Xinyu Fang, Haodong Duan, Dahua Lin, Jiaqi Wang, Yuhang Zang Categories: cs.CV

The Gap

Existing multimodal benchmarks fall into two camps. Full-state benchmarks (e.g., vision-language navigation, robotic manipulation) expose the entire environment at every step — the agent only needs the current observation, a Markov assumption. Episodic recall benchmarks (e.g., video QA, memorization tasks) test memory after an episode ends, not during closed-loop decision-making. Neither isolates the core challenge: reconstructing past observations from partial feedback while acting in real time.

Prior work like MM-Nav or RoboVQA conflates memory with other skills (planning, object recognition, motor control). There’s no controlled way to measure when the agent forgets and why it fails. This paper plugs that hole.

[ASCII logic topology showing: Problem -> Assumption -> Method -> Evidence -> Conclusion]

    Problem: MLLMs fail in partially-observable environments.
       |
       v
    Assumption: Failure stems from poor memory, not other skills.
       |
       v
    Method: RNG-Bench (Matching Pairs + 3D Maze).
        / \
       /   \
   Difficulty    Memory Gap
   Axes          Metric
       \   /
        v v
    Evidence: Models saturate on easy, fail on hardest
    (128K tokens, 350 images).
       |
       v
    Conclusion: Memory is the bottleneck; fine-tuning on
    optimal rollouts alleviates it.

The Increment

One sentence: Before this paper, no benchmark could isolate a multimodal agent’s memory reconstruction ability from its other capabilities; after this paper, researchers have a controlled, interpretable testbed to measure and improve that ability.

Core Mechanism

RNG-Bench consists of two complementary games, both designed to force the agent to remember observations that are no longer visible and then act based on that memory.

Matching Pairs is a grid of face-down cards. The agent can flip two adjacent cards at a time. If they match, they stay face-up; otherwise, they flip back. The agent must remember card identities and their locations across multiple flips. The difficulty is controlled by grid size (number of pairs), visual pattern (distinctness of card faces), and observation modality (partial vs. full view of the flipped pair).

3D Maze is a first-person navigation task. The agent sees only an egocentric view (a narrow cone). It must build a spatial map of corridors and dead ends while moving. At certain points it must report directions to goals it has seen earlier. Difficulty axes: maze size, texture richness, and field-of-view.

Both games run under a unified harness with three controlled axes: grid size / maze size, visual pattern complexity, and observation modality (e.g., RGB vs. depth vs. sketch). The harness also introduces a head-to-head duel protocol: each test instance is paired with a matched control that differs only in the memory requirement, so instance-level variance (e.g., random card positions) is cancelled out. The Memory Gap metric subtracts the score achieved with perfect memory from the actual score, isolating forgetting from poor action selection.

[ASCII diagram of method internals: components, data flow, operations]

              +-------------------+
              |   RNG-Bench       |
              |   (Unified Harness)|
              +--------+----------+
                       |
          +------------+------------+
          |                         |
   +------v------+          +------v------+
   | Matching    |          | 3D Maze     |
   | Pairs       |          | (Navigation)|
   +------+------+          +------+------+
          |                         |
          |  Controlled Axes:       |
          |  - Size (#pairs /       |
          |    maze steps)          |
          |  - Visual pattern       |
          |  - Observation modality |
          +------------+------------+
                       |
                       v
          +----------------------------+
          |  Evaluation Protocol       |
          |  - Head-to-head duel       |
          |  - Memory Gap = score_diff |
          +----------------------------+
                       |
                       v
          +----------------------------+
          |  Target: MLLMs (Qwen, GPT, |
          |  LLaVA, etc.)              |
          +----------------------------+

Structural Metaphor: Think of the MLLM as a detective solving a case in a dark house. Each room (observation) is only visited once, and the detective must reconstruct the floor plan (spatial memory) and remember where suspicious objects were (card identities). The benchmark is the “crime scene designer” who controls how many rooms (grid/maze size), how distinct the objects are (visual pattern), and whether the detective has a flashlight (observation modality). The Memory Gap is like a polygraph that separates “the detective never saw the clue” from “the detective saw it but forgot”. The head-to-head duel is like giving two identical houses but switching the floor plan order — controlling for the detective’s general cleverness.

Key Concepts

• Non-Markov property: The current observation alone does not contain enough information to act optimally. The agent must remember past observations. Example: In Matching Pairs, after flipping two cards and seeing they don’t match, the agent must remember the identities for later turns. If it only sees the current pair, it has no memory of previous pairs — a Markov policy would be stuck.

• Memory Gap metric: Let $L$ be the optimal score given perfect recall, and $S$ be the actual score. Memory Gap = $L - S$. This separates forgetting (gap caused by imperfect recall) from poor decision-making (gap caused by bad action selection even with perfect memory). For Matching Pairs, perfect memory means always matching known pairs first. For 3D Maze, perfect memory means having a full map. The gap is computed by comparing the agent’s trajectory to a reference that uses the same decisions but with ground-truth memory injected.

• Head-to-head duel protocol: To control for instance-level variance (e.g., card positions, maze layout), each test instance is paired with a “shadow” instance that differs only in the memory requirement. For Matching Pairs, the shadow instance might show all cards face-up (fully Markov) while the test instance only shows flipped pairs. The difference in performance between the two isolates the memory component. This is critical because a difficult card pattern could make both versions hard, but the duel subtracts that difficulty.

Framework Shift

Before (mainstream approach):        After (this paper):
+-----------------------+           +-----------------------+
| Full-state Markov     |           | Partial observations   |
| benchmarks (e.g.,     |           | with controlled memory |
| navigation with map)  |           | requirement            |
+----------+------------+           +----------+------------+
           |                                   |
           v                                   v
Agent uses current observation -> Agent must reconstruct past
           |                                   |
           v                                   v
No memory isolation ->                 Memory Gap metric ->
performance is confounded              forgetting isolated

One sentence: From evaluating MLLMs in fully-observable settings with no memory pressure to evaluating them in partially-observable settings where memory is the sole bottleneck, the core shift is controllable, measurable memory reconstruction.

Expert Assessment

Problem choice: Real gap. As MLLMs move into closed-loop robotics and games, memory becomes critical. Existing benchmarks conflate it with other skills, making it hard to diagnose. This fills a clean slot.

Method maturity: Clever insight. The two games are simple but the controlled axes (size, pattern, modality) and the duel protocol show careful experimental design. The Memory Gap is a nice formalism. However, the games are narrow — Matching Pairs is essentially a combinatorial memory task, and 3D Maze is a small grid. Scaling to richer environments (e.g., partially-observable 3D scenes with moving objects) would need more work.

Experimental integrity: Baselines are fair. They test many open-source and close-source MLLMs (Qwen, GPT-4o, LLaVA, etc.). The head-to-head duel controls variance well. One red flag: fine-tuning Qwen3.5-9B on optimal rollouts might use ground-truth memory during training, which leaks information. They should clarify whether the training data includes the memory state or just observation-action pairs. Also, the Memory Gap metric assumes the agent’s actions are deterministic given memory — but MLLMs are stochastic, so the gap might be noisy.

Writing quality: Clear and well-structured. The abstract and introduction are crisp. The section on fine-tuning is a bit rushed — they don’t discuss hyperparameters or whether the improvement transfers to out-of-distribution difficulty axes. Rewriting the fine-tuning section with ablation studies would elevate the paper.

Verdict: Weak accept — important benchmark but limited scope; the Memory Gap metric and duel protocol are the main contributions, not the specific games.

Takeaways

1. Use the Memory Gap metric in your own agent evaluations. It’s a principled way to separate forgetting from decision-making. You can compute it by running your agent normally and then again with oracle memory injected (e.g., by providing ground-truth state at each step). The difference is the memory gap.

2. The head-to-head duel protocol is a general technique for controlling instance-level variance in benchmarks. If you have two versions of a task that differ only in memory requirement (e.g., with and without a history panel), you can cancel out nuisance factors.

3. Fine-tuning on optimal rollouts (teacher forcing with ground-truth actions) works, but the paper doesn’t explore whether it causes the agent to overfit to the benchmark’s specific structure. Be cautious about transferring this without testing on unseen task variants.

4. Hardest configuration (128K tokens, 350 images): This is a pressure test for MLLM context windows. If you’re building a memory-augmented agent, this benchmark gives you a concrete upper bound of how many tokens your model can handle before forgetting.