MotionForesight: Turning Video Priors into 3D Motion Predictors with a Lightweight Adapter

Paper: 2607.16192 Authors: Homanga Bharadhwaj, Yash Jangir Categories: cs.CV

The Gap

Here’s the problem: robots and embodied agents need to predict how objects will move when someone interacts with them — a drawer will slide open, a lid will rotate, a cup will rise. Existing research splits into two camps, neither fully satisfying.

Camp 1 — Pixel-level video prediction (e.g., Sora-style models, diffusion video predictors): These models already encode rich priors about object motion from millions of videos. But they predict *pixels, not geometry. You get a plausible-looking future frame but no explicit 3D trajectories an agent can plan with. The knowledge is trapped in appearance.

Camp 2 — 3D scene flow / motion forecasting (e.g., point cloud trackers, depth-supervised methods): These give you explicit 3D motion, but they require depth sensors, 3D annotations, or rigid-body assumptions. They struggle with novel objects, deformable surfaces, and diverse environments.

The gap: no one has cleanly piped the rich motion priors in video models into explicit 3D geometric predictions without heavy supervision. You either get pixels-without-geometry or geometry-without-priors.

The logical path:

Problem: Embodied agents need future 3D object motion
   |
   v
Assumption: Video models already "know" how objects move
   |          (but output pixels, not 3D)
   |
v
Method: Repurpose video priors via lightweight adapter
   |     + pseudo-ground-truth from retrospective tracking
   |
   v
Evidence: 40k videos beats 1M+ video baselines
   |       on OOD objects, environments, viewpoints
   |
   v
Conclusion: Video priors -> explicit geometric forecasts
             is a viable and efficient pipeline

The Increment

One sentence: Before this paper, getting 3D motion predictions from video required either heavy 3D supervision or brute-force scale; after this paper, a lightweight adapter on a frozen video model can do it with 40k clips and no depth sensors.

Core Mechanism

MotionForesight has three stages, and the key trick is that the expensive components are frozen — only a small adapter learns.

Stage 1 — Retrospective dense tracking. Take a complete video clip of human-object interaction. Run a pretrained video model with a dense 3D tracker (built on top of it) to extract 3D trajectories for every point on the manipulated object. This gives you pseudo-ground-truth: you know where every point *was at every timestep, including the future. But this only works after the fact — you needed the full clip.

Stage 2 — Mask latent encoding. Instead of asking the model to predict future RGB frames or depth maps (expensive, noisy), MotionForesight encodes the object’s future appearance and geometry into compact learned *mask latents. These are low-dimensional codes that represent “what the object looks like and where it is” without reconstructing full pixels. The video model and tracker are frozen here too.

Stage 3 — Forward prediction via adapter. Now the critical step: train a lightweight adapter network that takes the observed frames only (no future information) and the retrospective tracking representation from those frames, then predicts the mask latents for future timesteps. The adapter learns to turn “what I’ve seen so far in track space” into “what the object will do next.” The entire backbone stays frozen — only the adapter trains.

Data flow:

[Observed Frames] --> [Frozen Video Model] --> [Frozen Dense Tracker]
                                                      |
                                                      v
                                              [Track Representations]
                                                      |
                   [Mask Latent Encoder] <---  (from complete clips,
                        |                   pseudo-ground-truth)
                        v
                  [Mask Latents (future)]
                        |
                        v
              [Lightweight Adapter (LEARNED)]
                        |
                        v
              [Predicted Future 3D Trajectories]

The structural metaphor — a weather forecaster learning from old almanacs. Imagine you’re training a new weather forecaster. You have a brilliant but expensive meteorologist (the pretrained video model) who can perfectly explain *past weather patterns — given a full week of data, they can draw detailed maps of where every air mass went. But they can only work backwards, after the fact.

You also have a giant archive of old weather almanacs (the 40k training videos). For each past storm, your expensive meteorologist has already drawn the complete trajectory map (pseudo-ground-truth tracks).

Now you hire a junior intern (the lightweight adapter). The intern’s job: look at only the first two days of a storm, study the trajectory maps the senior meteorologist drew for similar past storms, and predict what happens on days 3–7. The intern doesn’t need to understand fluid dynamics from scratch (that’s in the senior’s frozen knowledge). They just need to learn the pattern of “given this opening, here’s the likely continuation.”

The mask latents are like the intern’s shorthand notes — not a full weather map, but a compact code saying “the high-pressure ridge shifts northeast and weakens.” Cheaper to predict, sufficient for planning.

This is why it works with so little data: the expensive knowledge (how objects move during interaction) is already baked into the frozen video model. The adapter only learns the much simpler mapping from “retrospective understanding” to “forward prediction.”

Key Concepts

• Pseudo-ground-truth from retrospective tracking: You can’t easily label 3D point trajectories in the future — that’s the whole prediction problem. But you *can run a tracker on a complete video after it’s been recorded to get full trajectories. This is the “cheating” trick: use future information to generate training labels, then train a model that only sees past information. It’s like studying answer keys to learn how to solve problems. The risk is label quality — if the tracker is noisy, the forecaster inherits that noise. The paper argues the pretrained video model’s tracker is good enough.

• Freezing the backbone, learning only the adapter: In transfer learning, you can fine-tune everything (expensive, overfits on small data) or freeze everything and train a small new piece. MotionForesight takes the freeze approach. The video model and tracker have millions/billions of frozen parameters encoding “how the visual world works.” The adapter has far fewer parameters and only learns “how to turn retrospective tracks into predictions.” This is why 40k videos suffice — the adapter has a much simpler job.

• Scene flow vs. optical flow: Optical flow tells you how *pixels move between two frames — it’s 2D and appearance-based. Scene flow tells you how 3D points in the real world move — it’s geometric. The paper’s goal is future scene flow: not “where did the pixels go” but “where will the 3D points on this object be.” This distinction matters for robotics because an agent needs to know the physical trajectory, not just the visual appearance.

Framework Shift

Before (mainstream):                    After (this paper):

[Video Model]                           [Video Model] (frozen)
     |                                       |
     v                                       v
[Predict Future Pixels]                [Dense 3D Tracker] (frozen)
     |                                       |
     v                                       v
[No explicit 3D info]                  [Retrospective Tracks]
                                            |
                                            v
                                      [Mask Latent Adapter] (learned)
                                            |
                                            v
                                      [Future 3D Scene Flow]

From “video models as pixel generators” to “video models as motion-prior banks you can query for geometry,” the core shift is treating pretrained video understanding not as the end product but as a frozen oracle whose knowledge can be redirected through a small learned interface.

Expert Assessment

Problem choice: This is a real and important gap. The disconnect between video prediction (flourishing, well-funded) and embodied motion forecasting (data-hungry, underfunded) has been a known pain point. Connecting them is a natural and overdue idea. It sits at the intersection of video understanding and robotics — a hot trajectory in 2025-26.

Method maturity: The insight is clever — repurposing retrospective tracking as pseudo-labels is elegant and has precedent in self-supervised learning (e.g., using past frames as future targets). The mask latent trick to avoid predicting full RGB is sensible. However, the adapter itself is relatively simple, and the heavy lifting is done by the pretrained video model’s quality. If the backbone tracker is poor, this whole pipeline degrades. There’s an implicit dependency on upstream model quality that the paper could stress-test more rigorously.

Experimental integrity: The baselines are reasonably fair — they compare against models trained on 1M+ videos and show competitive or better performance with 40k, which is a strong result. The OOD generalization experiments (novel objects, environments, viewpoints) are well-designed. One concern: the pseudo-ground-truth evaluation depends on the same tracker used for training — there’s a circularity risk where the model is optimized to match its own label generator. Real-world robotic validation (did the predicted trajectory help a robot grasp?) would strengthen the claims significantly.

Writing quality: The paper reads clearly and the structure is logical. The ablation studies could be deeper — specifically, how sensitive is performance to tracker quality, adapter size, and the number of training videos? The related work section could better position against recent 3D reconstruction-from-video methods (e.g., DUSt3R family) that also extract geometry from video models. That comparison would sharpen the contribution.

Verdict: weak accept — The core idea of repurposing video priors for 3D motion forecasting is genuinely useful and the results are promising, but the evaluation stops short of demonstrating real embodied impact, and the method’s dependence on upstream tracker quality needs more scrutiny.

Takeaways

1. The “freeze backbone, learn interface” pattern transfers broadly. Anywhere you have a large pretrained model that encodes useful priors in the wrong output format, a lightweight adapter can redirect it. This applies to language models (repurposing for structured prediction), audio models (repurposing for music generation), etc.

2. Retrospective pseudo-labels from complete sequences. If you have access to full sequences at training time but only partial observations at test time, you can generate pseudo-ground-truth from the full data and train on partial input. This is a general self-supervised recipe.

3. Mask latents as a compression trick for prediction targets. Instead of predicting high-dimensional outputs (full point clouds, RGB frames), learn a compact latent code for the target and predict that instead. This reduces the prediction burden and can improve generalization. The idea of “predict the code, not the signal” has legs in many forecasting tasks.