OmniAgent: Native Active Perception as Reasoning for Long Video Understanding

Paper: 2606.19341 Authors: Zhenghao Xing, Ruiyang Xu, Yuxuan Wang, Jinzheng He, Ziyang Ma, Qize Yang, Yunfei Chu, Jin Xu, Junyang Lin, Chi-Wing Fu Categories: cs.CV, cs.CL, cs.SD

The Gap

Existing long video understanding models fall into two camps. Passive models (e.g., Video-LLaVA, LLaVA-NeXT-Video) process every frame uniformly — they “watch it all” then answer. Their compute cost scales linearly with video length, and they waste resources on irrelevant segments. Interactive models (e.g., VideoAgent, Agentic Video) can query specific clips, but they rely on a global pre-scan that still processes the entire video upfront. The context bottleneck remains: memory or cost still scales with total duration.

The core blind spot: no model treats video understanding as an inherently partial-observable problem. A human watching a long video doesn’t store every frame; they form a running summary and query specific moments on demand. The paper formalizes this as a POMDP (Partially Observable Markov Decision Process) — the agent only has a limited textual memory, and must actively decide which audio-visual snippets to extract next.

Problem:          Video understanding cost scales with length
                     |
Assumption:      Must pre-scan or uniformly process all frames
                     |
Method:          Iterative Observation-Thought-Action cycle (POMDP)
                     |
Evidence:        10 benchmarks; 7B model beats 72B model on LVBench
                     |
Conclusion:      Active perception decouples reasoning from raw duration

The Increment

One sentence: Before this paper, video understanding was a “watch everything, then answer” pipeline; after it, understanding is an active, turn-by-turn reasoning process where the agent chooses what to perceive next based on its current belief state.

Core Mechanism

OmniAgent operates as a turn-based POMDP agent. At each turn $t$, the agent maintains a persistent textual memory $M_t$ (a running summarization of everything observed so far). The cycle has three steps:

1. Observation: The agent perceives $M_t$ (and optionally the raw video timestamp of the current action) to understand its current belief.
2. Thought: A reasoning step that decides *what information is missing to answer the original question. This generates a query action — e.g., “Look at the segment from 3
   to 3
   ” or “Listen to the audio at 5
   ”.
3. Action: The system executes the query on the raw video/audio, extracts the requested snippet, and updates the memory $M_{t+1}$ with the new information (e.g., a text description of the segment).

The cycle repeats until a stopping condition (e.g., confidence threshold or max turns). The final answer is derived from the accumulated memory.

To train this agent, two novel mechanisms are introduced:

• Agentic Supervised Fine-Tuning (Agentic SFT): To bootstrap the agent’s ability to generate useful actions, the authors synthesize a training set via *best-of-N trajectory sampling: for each question, they roll out many possible action sequences, score them by the quality of the final answer, and select the best trajectories. Dual-stage quality control (first filter by answer correctness, second by coverage of relevant segments) ensures high-quality demonstrations.
• Agentic Reinforcement Learning with TAURA: After SFT, the agent is further fine-tuned with RL to improve exploration. TAURA (Turn-aware Adaptive Uncertainty Rescaled Advantage) adjusts the reward credit assignment by measuring the *entropy of the policy at each turn. Turns where the agent is highly uncertain (high entropy) — i.e., discovery turns — get larger reward bonuses. This prevents the RL from ignoring the early, crucial information-seeking actions.

           +-----------+
           |  Initial  |
           |  Question |
           +-----+-----+
                 |
                 v
  +--------------------------+
  |  Observation (Memory M)  |
  +------------+-------------+
               |
               v
  +--------------------------+
  |  Thought (Query needed?) |
  +------------+-------------+
               |
               v
  +--------------------------+
  |  Action (Extract segment)|
  +------------+-------------+
               |
               v
  +--------------------------+
  |  Update Memory -> M'     |
  +--------------------------+
        |                      \
        v                       v
   Stop if confident      Continue loop
   else repeat              (max T turns)

Structural metaphor: Detective Cold Case

Imagine a detective (agent) re-opening a cold case (video). The detective has a case folder (textual memory) that starts nearly empty — just the initial question: “Who killed Mr. X at the party?”

• Observation: The detective reviews the folder. It contains a few notes from the initial briefing, but much is missing.
• Thought: The detective thinks: “I need to know the time of death. The autopsy report isn’t in the folder, but the coroner’s interview might be available.” This is a decision about *what to look up next.
• Action: The detective drives to the coroner’s office, retrieves the interview transcript, and writes a summary into the folder. The folder now has more information.

The detective repeats: reads folder, thinks of next missing piece (e.g., “Did anyone have a motive?” → checks security footage timestamp), takes action (watches a specific 2-minute clip), updates folder. Over several turns, the folder becomes a coherent case summary that answers the question without ever reviewing every second of the party video. The detective never watches the entire party — only the snippets logically needed.

Similarly, OmniAgent never processes the full video. It only extracts the small segments that the reasoning chain requires. The reasoning cost grows with the number of turns (which is bounded by complexity), not with the raw duration.

Key Concepts

• POMDP (Partially Observable Markov Decision Process): A formal framework for decision-making under uncertainty where the agent cannot directly observe the full state of the world. It can only see observations (here, the textual memory) and must infer the hidden state (the full video content). The agent’s actions actively modify the observation stream. Example: In a game of poker, you don’t see opponents’ cards; you bet based on your hand and their actions. Similarly, OmniAgent doesn’t see all frames; it bets on which segment to query next based on its accumulated memory.

• Test-Time Scaling: The property that a model’s performance improves as it is allowed more computation at inference time. Most models saturate or degrade with more compute (e.g., running twice as many tokens just adds noise). OmniAgent shows that more reasoning turns → better answers, confirming the agent is using additional turns to genuinely gather necessary information, not just hallucinate. This is analogous to a student who gets more time to read source material: they answer more accurately.

• TAURA (Turn-aware Adaptive Uncertainty Rescaled Advantage): A credit assignment trick for RL. In typical RL for agents, rewards for correct final answers are distributed equally among all actions. But in the video agent, early actions (like “look at timestamp 1

  ”) are often critical but happen long before the final reward. TAURA measures the policy entropy at each turn: high entropy means the agent is uncertain, often indicating a pivotal discovery step. It rescales the advantage (reward bonus) to give higher credit to those high-entropy turns. Example: If the agent is very uncertain whether to check video or audio (both equally plausible), that turn’s action probably shaped the rest of the trajectory, so it deserves more credit.

Framework Shift

Before (mainstream approach):        After (this paper):
All frames (or uniform sample)       Iterative active query
+--------+   +--------+   +-------+    +-------+   +-------+   +-------+
| Frame1 |...| Framen |   |Answer |    | Query1|-->| Query2|-->|Answer |
+--------+   +--------+   +-------+    +-------+   +-------+   +-------+
   |               |          ^            |           |            ^
   v               v          |            v           v            |
+---+           +---+         |        +-------+   +-------+       |
|Fwd|           |Fwd|---------+        |Memory|-->|Memory|--------+
+---+           +---+                  +-------+   +-------+
(compute per frame)                    (compute per query)

Cost = O(video length)                 Cost = O(reasoning turns)

From passive, uniform ingestion to active, query-driven reasoning — the core shift is decoupling understanding cost from raw video duration.

Expert Assessment

Problem choice: A real and growing gap. As videos get longer (hour-long lectures, surveillance, movies), linear scaling becomes prohibitive. Passive models haven’t addressed this; interactive models still process everything once. The POMDP framing is elegant — it treats video understanding as an inherently information-seeking problem.

Method maturity: Clever, not brute-force. The best-of-N trajectory synthesis is practical (though expensive), and TAURA is a neat insight to fix credit assignment. However, the RL training pipeline is complex and may be brittle to hyperparameters. There’s no comparison to simpler alternatives (e.g., just querying random segments), so we can’t fully attribute gains to the active perception.

Experimental integrity: Impressive empirical scope — 10 benchmarks, including VideoMME, LVBench, etc. The 7B model beating Qwen2.5-VL-72B on LVBench is striking. Ablations on turn limit and TAURA are present. One concern: LVBench is a Chinese benchmark; the authors are Chinese; could there be subtle dataset bias? Probably not fatal. But the paper lacks cost analysis — how many tokens/dollars does 10 turns cost vs processing a 1-hour video?

Writing quality: Well-structured overall. The POMDP formulation is clearly motivated. However, the “Omni-Modal” claim is thin — only audio and vision are used (no tactile, olfactory, etc.). The section on TAURA could be more intuitive; a reader without RL background will struggle. Rewriting the credit assignment explanation with a concrete example would elevate the paper.

Verdict: strong accept — The active perception framing and the decoupling result are significant contributions, backed by solid benchmarks. The test-time scaling property is particularly promising for future research.

Takeaways

1. Decoupling strategy via POMDP: Any problem that involves processing a large, partially observable stream (long documents, multi-hour audio, surveillance feeds) can be recast as an active query process. The key is to maintain a compact textual memory and design action space that extracts small, relevant chunks.

2. Turn-level uncertainty for credit assignment: In multi-turn RL, using entropy to weight rewards is a general idea. It can be applied to other multi-step reasoning tasks (e.g., multi-hop QA, code debugging chains) where early steps are critical but temporally distant from reward.

3. Best-of-N trajectory synthesis for agent bootstrapping: When you can cheaply simulate many rollouts (e.g., query a video with random actions and evaluate the final answer via an oracle model), you can generate high-quality demonstrations without manual annotation. This technique transfers to any environment with a simulator and a reward oracle.

4. Test-time scaling as a validation metric: If your model shows positive test-time scaling, it’s a strong indicator that the model is *actually using additional compute to gather information, not just hallucinating or being wasteful. Use this as a sanity check for any iterative model.