YoCausal: Testing Whether Video Models Understand Causality or Just Memorize Time

Paper: 2605.30346 Authors: You-Zhe Xie, Yu-Hsuan Li, Jie-Ying Lee, Kaipeng Zhang, Yu-Lun Liu, Zhixiang Wang Categories: cs.CV

The Gap

Video diffusion models are being positioned as “world models” — systems that understand how the world works, not just how pixels move. Existing benchmarks test them on synthetic data (bouncing balls, block towers), which creates a sim-to-real gap. More critically, these benchmarks conflate two distinct abilities: perceiving temporal direction (does this look like it’s playing forward?) and understanding causality (does this action cause that outcome?). A model might excel at the former by memorizing statistical patterns while failing at the latter.

The paper identifies that no existing benchmark separates these two capabilities using real-world data at scale. Prior work either uses expensive human annotations or synthetic scenarios that don’t transfer.

Problem: VDMs claim world model status
   |
   v
Assumption: Temporal coherence ≠ causal understanding
   |
   v
Method: Reverse real videos (zero-cost counterfactuals)
   |    + VLM stratification (causal vs non-causal)
   v
Evidence: 13 VDMs tested
   |    RSI (arrow-of-time) vs CCI (causality) diverge
   v
Conclusion: Time perception =/=> causal reasoning

The Increment

One sentence: Before this paper, we assumed video models that handle temporal coherence understand causality; after, we have evidence they’re pattern-matching time’s arrow without grasping cause-effect.

Core Mechanism

YoCausal operates on two levels. Level 1 measures the Reverse Surprise Index (RSI): take a real video, reverse it temporally, feed both versions to the video diffusion model, and compare denoising losses. If the model is surprised by the reversed version (higher loss), it perceives the arrow of time. This requires no labels — just the model’s own loss function.

Level 2 introduces the Causality Cognition Index (CCI). Here’s the trick: not all videos contain visible causality. A person walking forward vs backward might look equally plausible (non-causal), but an egg breaking vs un-breaking is physically impossible (causal). The paper uses a Vision-Language Model (VLM) to automatically classify videos into causal and non-causal subsets based on whether reversing them violates physical laws. Then it computes RSI separately for each subset. The gap between causal-RSI and non-causal-RSI reveals whether the model truly understands causality or just detects temporal bias.

Input: Real video V
   |
   +---> Forward V ---> VDM ---> Loss_fwd
   |
   +---> Reverse V ---> VDM ---> Loss_rev
   |
   v
RSI = (Loss_rev - Loss_fwd) / Loss_fwd
   |
   v
VLM: Is reversing V physically impossible?
   |
   +---> Yes: Causal subset ---> RSI_causal
   |
   +---> No: Non-causal subset ---> RSI_non-causal
   |
   v
CCI = RSI_causal - RSI_non-causal

Think of it like testing a chef’s understanding of cooking. Level 1 asks: can you tell when a video of cooking is played backward? A good chef should notice the egg un-scrambling. But here’s the catch — even someone who’s never cooked might notice because backward motion looks weird in general. Level 2 is the real test: show them two videos — one of a person stirring (reversible, no causality) and one of an egg breaking (irreversible, causal). A true chef will be much more surprised by the reversed egg than the reversed stirring, because they understand the causal process of protein denaturation. Someone just pattern-matching “backward looks weird” will be equally surprised by both.

Key Concepts

• Violation of Expectation (VoE): A paradigm from infant cognition research. Show a baby something physically impossible (like an object passing through a solid wall), and they stare longer — they’re surprised. YoCausal adapts this: if a model “expects” forward causality, reversing it should increase prediction error. The beauty is you don’t need to teach the model what’s impossible; you just measure its surprise.

• Arrow of Time vs Causality: The arrow of time is statistical — entropy increases, milk diffuses into coffee. Causality is mechanistic — striking a match causes ignition. A model can learn “things usually happen in this order” (arrow of time) without understanding “this action produces that effect” (causality). YoCausal’s two-level design separates these: RSI measures the former, CCI isolates the latter.

• Zero-Cost Counterfactuals: In causal inference, you need counterfactual samples (what would happen if X didn’t occur?) to test understanding. Generating these is expensive. But temporal reversal gives you counterfactuals for free: the reversed video is a world where causality runs backward. If the model understands causality, it should reject this impossible world more strongly than it rejects non-causal reversals.

Framework Shift

Before (mainstream approach):        After (this paper):

Synthetic data                       Real-world videos
    |                                    |
    v                                    v
Test temporal coherence          Temporal reversal (free)
    |                                    |
    v                                    v
Single metric                        Two-level hierarchy
(conflates time + causality)         RSI: arrow of time
                                     CCI: true causality
                                         |
                                         v
                                     VLM stratification
                                     (causal vs non-causal)

From synthetic-single-metric to real-hierarchical-disentangled, the core shift is separating what models perceive from what they understand.

Expert Assessment

Problem choice: This is a real gap. The field has been sloppy about equating temporal modeling with causal reasoning. The VoE paradigm is a clever import from cognitive science, and the observation that temporal reversal provides free counterfactuals is genuinely useful. The problem sits at a critical juncture — as VDMs scale toward world models, we need to know if they’re learning physics or just statistics.

Method maturity: The core insight (reversal as counterfactual) is elegant. The VLM stratification is pragmatic but introduces a dependency — if the VLM misclassifies videos, CCI becomes noisy. The paper doesn’t deeply validate the VLM’s causal/non-causal labels against human judgment. Also, denoising loss as a proxy for “surprise” assumes the model’s training objective aligns with causal understanding, which is debatable. A stronger approach might include probing the model’s internal representations or testing intervention-based causality.

Experimental integrity: Testing 13 VDMs is thorough. The finding that RSI and CCI diverge is compelling — models score high on arrow-of-time but low on causality. However, the paper doesn’t control for confounds like motion complexity or object semantics. A reversed video of a complex scene might have higher loss simply because it’s harder to denoise, not because causality is violated. The baselines are fair, but the paper would benefit from ablations isolating these factors.

Writing quality: The abstract and introduction are strong. The method section buries the VLM stratification details — this is the linchpin of Level 2 and deserves more scrutiny. The results section shows aggregate metrics but lacks failure case analysis. Which types of causal events do models fail on? Mechanical (breaking), chemical (burning), biological (growing)? That breakdown would make the findings actionable.

Verdict: weak accept — The problem is important, the method is novel, and the findings challenge the field’s assumptions. But the VLM dependency and lack of confound analysis leave questions about how much of the CCI gap is truly causal vs artifactual.

Takeaways

For benchmark designers: Temporal reversal is a cheap, scalable way to generate counterfactual data. Pair it with automatic stratification (via VLMs or other classifiers) to disentangle different reasoning types. This pattern generalizes beyond video — reverse audio for speech models, reverse code execution traces for program synthesis.

For model developers: If your video model scores well on temporal coherence but poorly on CCI, you’re overfitting to statistical patterns. Consider augmenting training with explicit causal interventions (e.g., masking cause frames and predicting effects) or contrastive losses that penalize physically impossible sequences.

For researchers: The two-level hierarchy (perception vs understanding) is a transferable framework. Apply it to other domains: does a language model perceive grammar (Level 1) or understand semantics (Level 2)? Does a robot perceive object motion (Level 1) or understand affordances (Level 2)? The key is finding a cheap transformation (like reversal) that creates a counterfactual world where understanding, but not perception, should fail.