Separating Camera and Object Motion in Self-Supervised Video Understanding

Paper: 2607.21576 Authors: Lukas Knobel, Andrew Zisserman, Yuki M. Asano Categories: cs.CV

The Gap

In video understanding, a fundamental challenge is decomposing frame-to-frame changes into “camera motion” (the lens moving) and “object motion” (things moving in the scene). Mainstream self-supervised video learning methods (like video masked autoencoders or contrastive learning) often treat this entangled mix as a single, holistic “change” signal. They learn features that capture something is happening, but not what is causing it. This leaves the learned representations brittle and hard to analyze—like a translator who can tell you a sentence changed but not whether the subject, verb, or adverb changed. This paper targets that specific limitation: can we learn to separate these two sources of dynamics without expensive per-pixel supervision (like optical flow)?

+---------------------------+
| Problem: Video features   |
| entangle camera and       |
| object motion.            |
+-------------|-------------+
              |
              v
+---------------------------+
| Assumption: A pretrained  |
| frozen image ViT contains |
| sufficient information    |
| to disentangle them.      |
+-------------|-------------+
              |
              v
+---------------------------+
| Method: Structured        |
| Dynamics Model (SDM)      |
| explicitly separates      |
| dominant (camera) from    |
| residual (object) dynamics|
| via future prediction.    |
+-------------|-------------+
              |
              v
+---------------------------+
| Evidence: Trained on real |
| video + weakly supervised |
| synthetic data (Kubric).  |
| Beats baselines on new    |
| ProbeMotion benchmark.    |
+-------------|-------------+
              |
              v
+---------------------------+
| Conclusion: We can build  |
| structured video dynamics |
| models on top of frozen   |
| image backbones.          |
+---------------------------+

The Increment

One sentence: Before this paper, video representation learning conflated all motion; after it, we have a practical method to explicitly factor video dynamics into camera and object components using only self-supervised and weak supervision.

Core Mechanism

The Structured Dynamics Model (SDM) doesn’t process raw pixels. Instead, it starts with the frozen features from a pretrained Vision Transformer (ViT) for each frame of a video clip. Its core job is to predict the features of a future frame, but it does this in a structured way.

The model has two main pathways. The first pathway predicts the dominant dynamics. This captures the largest, most consistent changes across the video, which the authors argue are mostly due to camera motion (panning, zooming). The second pathway predicts the residual dynamics. This captures what’s left after subtracting the dominant prediction—typically the motion of objects within the scene. Both pathways take the current frame’s features and produce a prediction for the next frame. The final loss function encourages the sum of these two predictions to be close to the actual future frame’s features, while also using synthetic videos with known camera and object motion to provide weak, direct supervision on each pathway’s output.

[Input: Frozen ViT Features per Frame]
                 |
                 v
+----------------|----------------+
|                |                |
|  [Dominant Dynamics Pathway]    |
|  (e.g., learns global shifts)  |
|                |                |
|                +----> [Pred_A]--+
|                                 |
|  [Residual Dynamics Pathway]    |
|  (e.g., learns local changes)  |
|                                 |
|         +----> [Pred_B]--------+
|         |                       |
|         v                       |
|    +----|----+                  |
|    | Combine |                  |
|    | (Pred_A |                  |
|    | +Pred_B)|---> [Final Pred] |
|    +---------+                  |
|                                |
|    [Loss] = dist(Future_Feat,  |
|             Final_Pred)        |
+--------------------------------+

Think of this like a soccer coach watching a game film. The coach doesn’t see pixels; they see players (features). To predict the next moment, the coach first accounts for the camera panning (dominant dynamics): if the camera swings right, all players’ positions shift left in the frame. This is the bulk of the change. After mentally stabilizing the view (subtracting the pan), the coach focuses on player movements (residual dynamics): the striker running, the defender stepping up. SDM does the same—it first predicts the big, global shift (camera), then refines it with the finer, local movements (objects). The synthetic Kubric data is like the coach having access to a video game replay where you can control the camera and player actions separately, providing perfect practice for learning the decomposition.

Key Concepts

• Frozen Image Features: Instead of training a video model from scratch or fine-tuning an image model, they use a standard image ViT that is *completely locked. The weights don’t change during training. This is a powerful constraint: it forces all the learning about motion to happen in the new SDM modules layered on top. It asks, “How much can we understand about dynamics by just looking at how pre-trained, static image features change over time?” This isolates the dynamics learning problem.

• Structured Future Prediction: This is the key innovation. Instead of having one big model predict the next frame’s features (an “entangled” latent), or predicting a separate transition token for every pixel patch (computationally heavy and “unstructured”), they have two distinct predictors with a clear semantic division of labor: one for the dominant source of change (camera), one for the remainder (objects). The structure (the split) is the main contribution, providing a useful inductive bias.

Framework Shift

Before (mainstream approach):        After (this paper):
                                     
[Video Frames]                       [Video Frames]
      |                                    |
      v                                    v
[One Encoder]  -->  [Single Entangled    [Frozen Image Encoder]
      |              Motion Latent]            |
      |                                    [Split into Two Pathways]
      v                                    /      |
[Prediction /                          v         v
 Contrastive Loss]              [Dominant Path] [Residual Path]
                                      |         |
                                      v         v
                                   [Camera]  [Object]
                                      \       /
                                       v     v
                                   [Structured Pred.]

From learning a single, monolithic video representation to explicitly factoring dynamics into camera and object components, the core shift is from entangled to structured motion modeling.

Expert Assessment

Problem choice: This is a real, well-identified gap. The entanglement of camera and object motion is a known annoyance in many downstream tasks (action recognition, video generation, robotics). Framing it as a representation learning problem for *features rather than raw pixels is a smart and modern angle. It sits at the intersection of video self-supervised learning and geometric/compositional understanding.

Method maturity: It’s a clever insight with a clean design. Using two prediction pathways is an elegant way to impose structure. However, it’s more of a structured architectural prior than a fundamental algorithmic breakthrough. The reliance on synthetic data (Kubric) for the direct “camera/object” supervision signal is necessary but also a limitation—it assumes the synthetic dynamics are a good proxy for the real world. A simpler approach might not exist; the structure is the point.

Experimental integrity: The baselines are reasonable (global CLS, avg-pool, VGGT). The creation of the ProbeMotion benchmark is a significant contribution, allowing targeted evaluation. The numbers show clear gains over the baselines, especially for tasks requiring motion decomposition. One potential red flag: the comparison to VGGT (a strongly supervised model) is favorable in some probes but not all, and the gap isn’t always huge. It would be stronger if the real-world task performance gains were even more dramatic, but the results are solid and support the paper’s claims.

Writing quality: The paper is clear and well-structured. The motivation is crisp. The method section is understandable. Where it cuts corners is in the *depth of analysis. The ablation studies are okay, but a deeper dive into what each pathway actually learns (e.g., visualizing dominant vs. residual feature changes) would elevate it. The related work is standard but could more sharply position this against other decomposition approaches in computer vision.

Verdict: weak accept — The core idea of structured dynamics prediction via two pathways is novel and well-motivated. The experiments are convincing, and the ProbeMotion benchmark is a valuable resource. It’s not a field-shifting paper, but it’s a solid, incremental contribution that advances the state of structured video understanding.

Takeaways

The most stealable idea is the structured prediction framework for disentangling factors in temporal data. If you’re working on any problem where change has multiple, distinct sources (e.g., in audio, separating speaker and background noise; in robotics, separating arm movement from base movement), this “predict the dominant residual first, then refine with a secondary predictor” architecture is a transferable design pattern. The practice of leveraging synthetic data with known factors to provide weak supervision for a disentanglement task on real data is also a highly practical strategy for representation learning.