Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs

Paper: 2605.22823 Authors: Jongseo Lee, Hyuntak Lee, Sunghun Kim, Sooa Kim, Jihoon Chung, Jinwoo Choi Categories: cs.CV

The Gap

Video-LLMs like VideoChat, Video-LLaVA, and LLaVA-NeXT-Video excel at complex temporal reasoning—they can summarize plots, track events, answer “what happens after X?” Yet show them a ball moving left and ask “which direction?”, they score 25-30%, barely above random chance. The field assumed temporal understanding implies motion understanding. This paper reveals that’s false: models can track what and when without knowing which way.

Prior work focused on action recognition (what happened) and temporal localization (when it happened). Motion direction—a signed 2-D vector in image plane—was treated as a solved primitive. Existing benchmarks (ActivityNet, Kinetics) don’t isolate directional motion. When models fail, we don’t know if it’s the vision encoder, the projector, or the LLM’s answer binding.

Problem: Video-LLMs fail at basic motion direction (left/right/up/down)
   |
   v
Hypothesis: Motion signal exists but fails to bind to verbal answers
   |
   v
Method: Trace signal through pipeline + add projector-level motion objective
   |
   v
Evidence: Linear probes find signal at all stages; DeltaDirect fixes binding
   |
   v
Conclusion: Direction binding gap, not perception failure

The Increment

One sentence: Before this paper, Video-LLMs could describe temporal sequences but not reliably tell left from right; after, a projector-level motion objective (DeltaDirect) raises motion direction accuracy from 25.9% to 85.4% on synthetic data and improves real-world accuracy by 21.9 points without real-world training.

Core Mechanism

DeltaDirect operates at the projector level—the adapter between vision encoder and LLM. For adjacent video frames, it computes feature deltas (difference vectors in embedding space), then predicts a normalized 2-D motion vector (dx, dy) in image coordinates. This prediction is supervised with ground-truth optical flow, averaged over the frame and normalized to unit length.

The key insight: motion direction is a geometric signal that should be extracted before language binding. The projector already compresses spatial-temporal features; DeltaDirect adds a lightweight prediction head (two linear layers) that forces the projector to preserve directional information in a form the LLM can later decode. During training, this auxiliary loss runs alongside standard instruction tuning. At inference, the motion head is discarded—the projector’s learned representations now carry directional signal that the LLM can access through its normal text generation.

Frame t, Frame t+1
   |         |
   v         v
[Vision Encoder]  (CLIP/SigLIP)
   |         |
   +----+----+
        |
   [Feature Delta]  (subtract embeddings)
        |
        v
   [Projector]  (MLP adapter)
        |
        +---> [Motion Head] ---> predict (dx, dy)
        |                         compare to ground truth
        |                         backprop auxiliary loss
        v
   [LLM tokens] ---> "The object moves left"

Think of it like teaching someone to navigate by compass. The vision encoder sees the landscape (raw visual features). The projector is a guide who must compress the landscape into directions. Without DeltaDirect, the guide learns to describe landmarks (“there’s a tree, then a river”) but never learns to point north. DeltaDirect is compass training: for every pair of observations, the guide must explicitly state “we moved 30 degrees northeast.” After enough practice, the guide internalizes directional thinking—even when just describing landmarks, they now naturally encode which way things are. The compass itself (motion head) can be removed later; the directional habit remains in how the guide compresses information.

Key Concepts

• Directional Motion Blindness: Video-LLMs perform near chance (25-30% on 4-way classification) when asked to identify motion direction of a single object moving left/right/up/down, despite strong performance on complex temporal tasks. This isn’t a vision encoder failure—linear probes show motion direction is linearly separable from encoder features. It’s a *binding failure: the model has the signal but can’t connect it to the correct verbal answer token (“left” vs “right”).

• Direction Binding Gap: The paper traces motion direction information through the pipeline using linear probes. At every stage (vision encoder output, projector output, LLM hidden states), a linear classifier can recover motion direction with 70-90% accuracy. Yet the LLM’s final answer is near random. The gap: motion direction remains *accessible in the representation space but fails to bind to the correct answer option during text generation. It’s like having a compass reading in your pocket but answering “north” when you meant “south”—the information is there, but the connection to language is scrambled.

• Motion Concept Vector: Borrowed from concept activation literature, this is the direction in embedding space that maximally separates motion directions (e.g., “left” vs “right”). The paper measures concept vector magnitude across visual complexity levels. On simple synthetic videos (solid color backgrounds), the magnitude is strong. On real-world videos (cluttered scenes, camera motion), it weakens. This explains why synthetic instruction tuning doesn’t transfer: the concept vector learned from clean data is too weak to survive real-world noise. DeltaDirect’s explicit geometric supervision maintains stronger concept vectors across complexity levels.

Framework Shift

Before (mainstream approach):        After (this paper):

Video --> [Encoder] --> [Projector]  Video --> [Encoder] --> [Projector]
              |              |                     |         /    |    \
              v              v                     |    motion   |     |
          [LLM tokens]  "moves left"               |     head    |     |
                                                   v      |      v     v
          (hope motion direction                [Delta] -+   [LLM tokens]
           emerges from temporal                   |
           instruction tuning)                     v
                                              (dx,dy) loss
                                              
                                          (force projector to preserve
                                           geometric motion signal)

From implicit emergence to explicit geometric supervision, the core shift is: treat motion direction as a projector-level primitive that must be directly supervised, not as an emergent property of language-aligned temporal features.

Expert Assessment

Problem choice: Real gap. The field conflated temporal reasoning with motion perception. Action recognition benchmarks (Kinetics, ActivityNet) don’t isolate signed direction—“person walks left” vs “person walks right” are often the same action class. The authors constructed a minimal test (single object, four directions) that exposes the failure cleanly. This is good science: find the simplest case where the system breaks.

Method maturity: Clever and minimal. DeltaDirect adds ~2M parameters (two linear layers) and requires optical flow labels, but optical flow is cheap to compute or synthesize. The projector-level placement is smart—it’s the natural bottleneck where spatial-temporal compression happens. Alternative approaches (fine-tuning the vision encoder, adding motion tokens to the LLM) would be heavier and less modular. One concern: the method assumes adjacent-frame deltas capture motion, which breaks for fast motion or low frame rates. The paper doesn’t explore temporal stride as a hyperparameter.

Experimental integrity: Baselines are fair. The paper tests 8 Video-LLMs (VideoChat, Video-LLaVA, LLaVA-NeXT, etc.) and shows consistent failure. Linear probe analysis is rigorous—they control for prediction bias by measuring balanced accuracy. The synthetic→real transfer experiment is the key test: DeltaDirect improves real-world accuracy by 21.9 points *without real-world training data. That’s strong evidence the method learns transferable geometric structure, not dataset-specific shortcuts. One weakness: the real-world benchmark (MoDirect-RealBench) is only 400 samples. Larger-scale validation would strengthen claims.

Writing quality: Section 3 (diagnosis) is excellent—clear logic, good visualizations. Section 4 (method) is too brief. The motion head architecture (two linear layers, hidden dim, activation function) is buried in the appendix. The relationship between DeltaDirect loss weight and standard instruction tuning loss is underspecified. Rewriting Section 4 with an ablation table (loss weight, head architecture, temporal stride) would clarify design choices.

Verdict: weak accept — Identifies a real, overlooked failure mode and proposes a simple, effective fix with solid diagnostic work. The method is practical and the results are convincing, but the paper would benefit from deeper exploration of when and why the approach works (temporal stride, motion magnitude, camera motion).

Takeaways

• Diagnostic methodology: When a model fails, trace the signal through the pipeline with linear probes. If the signal is present at intermediate layers but absent in final outputs, the problem is binding, not perception. This technique transfers to debugging other modality gaps (audio-text, 3D-text).

• Projector-level objectives: For multimodal models, the projector is a leverage point. Adding lightweight auxiliary heads with explicit supervision (geometry, physics, causality) can force the projector to preserve structure that language alignment alone doesn’t capture. This is cheaper than retraining encoders or scaling LLMs.

• Synthetic→real transfer via geometric priors: DeltaDirect’s real-world transfer without real-world data suggests that geometric supervision (normalized motion vectors) learns more robust representations than language-only tuning. If your domain has geometric or physical structure (robotics, medical imaging, scientific video), consider explicit geometric losses even if your end task is language generation.