Video as Chain-of-Frame Reasoning

Paper: 2607.08763 Authors: Xinyan Chen, Ziyu Guo, Renrui Zhang, Dongzhi Jiang, Hongsheng Li Categories: cs.CV, cs.AI

The Gap

Prior work has shown that Large Language Models (LLMs) can reason using a step-by-step textual “Chain-of-Thought” (CoT). Separately, video generation models have become excellent at predicting plausible future frames from a starting image. The gap is that video generation models are trained on general video prediction, not on explicit reasoning. They can show what happens next, but they aren’t explicitly trained to show the logical steps of solving a problem visually. This paper asks: can we teach a video model to generate a sequence of frames where each frame is a logical reasoning step, forming a “Chain-of-Frame” (CoF)?

Problem: Video generators are trained for plausibility, not logical reasoning.
 |
 v
Assumption: A temporal sequence of frames can represent a reasoning chain.
 |
 v
Method: Create a reasoning video dataset (OpenCoF-17K) and fine-tune a video model (Wan-CoF).
 |
 v
Evidence: Wan-CoF outperforms its base model on 4 reasoning benchmarks.
 |
 v
Conclusion: Explicit temporal supervision and dedicated reasoning tokens improve "Chain-of-Frame" behavior.

The Increment

One sentence: Before this paper, video generation was a tool for visual prediction; after, it’s a potential medium for multi-step visual reasoning.

Core Mechanism

The framework operates in two main stages. First, they construct the OpenCoF-17K dataset. This isn’t just a collection of videos; it’s a dataset of “reasoning videos.” For each of 11 task families (like math, puzzles, logic), they provide: an initial frame (the problem), a final frame (the answer), and a sequence of intermediate frames that represent the step-by-step reasoning process. This teaches the model that a video isn’t just motion, but a logical narrative.

Second, they take a powerful existing video generation model, Wan2.2-I2V-A14B, and fine-tune it on this reasoning data, creating Wan-CoF. The critical innovation happens here. They don’t just retrain the whole model; they introduce special reasoning tokens. There are two types:

1. Visual Reasoning Tokens: These are embedded into the model’s latent space and are optimized to capture low-level, spatial details needed for the reasoning steps (e.g., the exact pixel change when drawing a line in a geometry proof).
2. Textual Reasoning Tokens: These are high-level semantic tokens, almost like a hidden script, that guide the overall logical flow and temporal structure of the video (e.g., “now apply the Pythagorean theorem”).

During generation, these tokens are injected at various points in the model’s processing (at different denoising steps and spatial locations), helping to coordinate the visual output into a coherent reasoning chain.

Input: [Initial Frame (Problem)] + [Task Instruction]
 |
 v
[Wan-CoF Model]
 |-- Embeds input into latent space.
 |-- Inserts [Visual Reasoning Tokens] for spatial detail.
 |-- Inserts [Textual Reasoning Tokens] for semantic flow.
 |-- Iteratively denoises through decoder layers.
 |
 v
Output: [Frame 1 (Step 1)] -> [Frame 2 (Step 2)] -> ... -> [Frame N (Answer)]

Think of it like a detective team solving a case on a whiteboard.

• The OpenCoF-17K dataset is the archive of past cases, each with a clear step-by-step evidence trail.
• The base Wan model is a talented artist who can draw any scene you describe, but has never solved a case.
• Wan-CoF is that artist *trained using the case archive.
• The Visual Reasoning Tokens are the specific forensic tools the artist uses (magnifying glass for details, red string for connections).
• The Textual Reasoning Tokens are the detective’s internal monologue and logic (“the suspect couldn’t have been here… therefore we check the alibi…”). During the investigation (generation), the artist uses both the tools and the monologue to draw each evidence board update (frame), leading logically to the solution (final frame).

Key Concepts

• Chain-of-Frame (CoF): Imagine explaining how to solve a Rubik’s cube. In text (CoT), you’d write: “Rotate the top face clockwise, then the right face counterclockwise…” In CoF, you’d show a video where each frame is the cube *after one move, with the sequence of frames being the complete solution. The reasoning isn’t described; it’s demonstrated across time. This paper’s core bet is that vision models can learn this demonstration.

Framework Shift

Before (mainstream approach):          After (this paper):
Text Chain-of-Thought:                 Visual Chain-of-Frame:
[Question] -> [LLM] -> [Text Steps] -> [Answer]
 |                                      |
 v                                      v
"Let's think step by step"             "Let's see it step by step"
 |                                      |
 v                                      v
[Word1, Word2, ...]                    [Frame1, Frame2, ...]

From reasoning in language to reasoning in vision, the core shift is representing intermediate logical states as images, not words.

Expert Assessment

Problem choice: Excellent. This is a real and timely gap. As multimodal models get stronger, the question of how they *reason visually is fundamental. Moving beyond text CoT is a natural and important frontier.

Method maturity: Clever insight, but the implementation is somewhat brute-force. Creating a specialized dataset and fine-tuning a massive model is effective but costly. The reasoning token mechanism is the clever part, providing an architectural handle for the problem. Simpler approaches (like just prompting models) clearly fail, so this heavier lift seems justified.

Experimental integrity: The gains over the strong baseline are convincing (significant improvements on 4 benchmarks). However, the baselines are fair: they compare against the exact same model before fine-tuning. A slight red flag: the paper explores more advanced token mechanisms (visual + textual) empirively but the primary results are from the fine-tuned model alone. The ablations are there, but more stress-testing would be ideal.

Writing quality: Very clear and well-structured. The “gap -> solution -> exploration” flow is logical. The methodology section could be more detailed on how the OpenCoF-17K data was curated and annotated – that process is the foundation of the work. Strengthening this section would elevate the paper’s rigor.

Verdict: strong accept — It introduces a compelling new paradigm (CoF) with a concrete first-step framework (OpenCoF) and demonstrates its viability, opening a clear path for future work.

Takeaways

1. Video as a Reasoning Medium: Stop thinking of video generation as just animation. It can be a structured, temporally-grounded reasoning process. This framing is highly transferable to robotics planning, procedural task tutorials, and scientific simulation.
2. Data is the New Bottleneck for Reasoning: The key enabling factor is the OpenCoF-17K dataset. The takeaway is that to build new capabilities, you often need to engineer new datasets that explicitly supervise that capability, not just scrape more generic data.
3. Token Specialization in Multimodal Models: The idea of injecting dedicated tokens (visual, textual) to steer different aspects of a model’s generation process is powerful. It’s a form of soft, learnable control that could be applied to other tasks like image editing or 3D generation to separate structure from texture or semantics from appearance.