Warp-as-History: Zero-Shot Camera Control for Video Generation via Warped Pseudo-History

Paper: 2605.15182 Authors: Yifan Wang, Tong He Categories: cs.CV

The Gap

Camera-controlled video generation has reached a point where models can generate videos following prescribed viewpoint trajectories. However, existing methods hit a wall: they either require expensive post-training on large-scale camera-annotated datasets (camera encoders, control branches, attention modifications), or they avoid training but pay the cost at test time through per-video optimization or extra guidance during denoising. The field lacks a method that achieves camera control without training, architectural changes, or test-time overhead.

Problem: Camera control = expensive training OR expensive inference
                |
                v
Observation: Video models already process visual history
                |
                v
Hypothesis: Camera motion ~ geometric transformation of history
                |
                v
Method: Warp past frames → feed as pseudo-history
                |
                v
Evidence: Zero-shot capability + 1-video LoRA generalizes
                |
                v
Conclusion: Camera control can be training-free interface problem

The Increment

One sentence: Before this paper, camera control required either large-scale training or test-time optimization; after, a simple warping interface unlocks zero-shot camera following in frozen models and strong generalization from one training video.

Core Mechanism

Warp-as-History treats camera control as a history manipulation problem. Given a target camera trajectory, the method takes past observed frames and geometrically warps them according to the camera motion (rotation, translation, zoom). These warped frames become “pseudo-history” — what the past would look like from the target camera’s viewpoint.

The key insight is that video generation models already have a visual-history pathway (they condition on previous frames). Instead of teaching the model about cameras through training, we pre-process the history to match the target viewpoint. Two critical operations make this work: (1) positional encoding alignment — the warped history tokens receive the positional encodings of the target frame being denoised, not their original positions, so the model treats them as spatially aligned with the generation target; (2) visible-token selection — tokens warped from regions without valid source observations (occluded areas, out-of-frame regions) are removed to avoid feeding the model garbage.

Input: Past frames [F_t-1, F_t-2, ...] + Target camera C_target
                |
                v
        Geometric Warp (rotation, translation, zoom)
                |
                v
        Warped frames [F'_t-1, F'_t-2, ...]
                |
                +---> Positional Encoding: Use target frame positions
                |
                +---> Visible Token Selection: Remove invalid regions
                |
                v
        Feed into model's visual-history pathway
                |
                v
        Output: Frame at target viewpoint

Think of it like a film editor’s trick. Imagine you’re making a movie and want to show a scene from a new angle, but you only have footage from the old angle. A naive approach would be to retrain the camera crew (expensive training) or reshoot the scene multiple times until it looks right (test-time optimization). Instead, Warp-as-History is like using perspective correction in post-production: take the existing footage, digitally transform it to approximate the new viewpoint, and feed it to the editing software as if it were shot from that angle. The software doesn’t know you cheated — it just sees “history” that happens to align with your target shot. The positional encoding alignment is like relabeling the footage’s metadata to match the new camera position, and visible-token selection is like masking out the parts where the transformation created artifacts or empty space.

Key Concepts

• Pseudo-History: In video generation, models typically condition on previous frames (history) to maintain temporal consistency. Pseudo-history is a manufactured version of this history — frames that didn’t actually exist at those viewpoints but are constructed to look like they did. Here, we create pseudo-history by warping real past frames according to camera motion. The model’s history pathway doesn’t distinguish between “real” history and “warped” history; it just processes visual tokens. By feeding warped frames with target-aligned positional encodings, we trick the model into generating content consistent with the target camera viewpoint. It’s like showing someone a photo taken from angle A, then asking them to imagine what angle B would look like — except we pre-warp angle A to approximate angle B before showing it.

• Positional Encoding Alignment: Video models use positional encodings to tell each token where it sits in space and time. Normally, a token from frame t-1 at position (x, y) gets encoding PE(t-1, x, y). When we warp that frame, the pixel moves to a new position (x’, y’), but we don’t want the model to think it’s still at (x, y) — that would create spatial misalignment with the target frame being generated. Instead, we assign it PE(t, x’, y’), the encoding of the target frame’s position. This alignment is crucial: it tells the model “this warped content belongs at this location in the frame I’m generating now,” enabling spatial coherence between pseudo-history and output.

• Visible-Token Selection: Geometric warping creates invalid regions — areas where the warp tries to pull information from outside the source frame’s boundaries or from occluded regions. If we feed these invalid tokens to the model, they contain noise or artifacts that corrupt generation. Visible-token selection identifies which warped tokens have valid source observations (pixels that actually existed in the original frame and weren’t occluded) and removes the rest. Think of it like a mask: after warping, we check each token’s provenance — if it came from a real pixel, keep it; if it came from extrapolation or empty space, discard it. This prevents the model from hallucinating based on garbage input.

Framework Shift

Before (mainstream approach):        After (this paper):

Camera params                        Past frames
     |                                    |
     v                                    v
Camera encoder                       Geometric warp
     |                                    |
     v                                    v
Control branch / Attention mod       Pseudo-history
     |                                    |
     v                                    v
Video model (trained)                Video model (frozen)
     |                                    |
     v                                    v
Output video                         Output video

Training: Large-scale dataset        Training: None (or 1 video)
Test-time: Standard inference        Test-time: Standard inference

From “teach the model about cameras through architecture and training” to “pre-process history to match the camera, let the model do what it already does.”

Expert Assessment

Problem choice: Real gap. Camera control is a legitimate bottleneck — existing methods either require prohibitive training data or sacrifice inference efficiency. The observation that video models already process history is obvious in hindsight but underexploited. This sits at a sweet spot: practical problem, elegant solution space.

Method maturity: Clever insight, not brute force. The core idea — warp history instead of training camera encoders — is simple and well-motivated. The two refinements (positional alignment and visible-token selection) are necessary, not ad-hoc. However, the method inherits limitations of geometric warping: it works for camera motion but not for complex scene dynamics or non-rigid deformations. The authors acknowledge this but don’t deeply explore failure modes.

Experimental integrity: Baselines are fair (training-based methods, training-free alternatives). The zero-shot results are impressive, and the 1-video LoRA generalization is the paper’s strongest claim. However, the experiments lean heavily on camera-annotated datasets (RealEstate10K, ACID). Real-world robustness (noisy camera estimates, dynamic scenes) gets less attention. The ablations are thorough for the method’s components but don’t stress-test assumptions (e.g., what happens when warping quality degrades?).

Writing quality: Clear and well-structured. The method section is crisp. The related work could be tighter — it lists many approaches but doesn’t sharply delineate why each fails to solve the problem. The discussion of limitations is honest but brief. Rewriting Section 4.3 (ablations) to lead with insights rather than tables would improve readability.

Verdict: weak accept — Solid contribution with a clean idea and strong empirical results, but the scope is narrower than the framing suggests (camera motion, not general viewpoint synthesis), and real-world robustness needs more scrutiny.

Takeaways

Reframe control as preprocessing: Instead of training models to understand control signals, preprocess inputs to match the desired output structure. This pattern transfers beyond camera control — think audio (warp spectrograms for pitch shift), 3D (warp point clouds for pose change), or text (reorder tokens for style transfer).

Exploit existing pathways: Video models already handle history; image models already handle spatial conditioning. Before adding new modules, check if existing pathways can be repurposed through clever input manipulation.

Positional encoding is a lever: Misaligning positional encodings with content is usually a bug, but here it’s a feature. When you warp or transform data, ask: should the positional encoding follow the content or stay fixed? The answer depends on whether you want the model to see “moved content” or “content at a new location.”

One-shot generalization via LoRA: The 1-video finetuning result is underappreciated. It suggests that lightweight adaptation on minimal data can teach models to generalize structural patterns (here, camera-history relationships) without overfitting. This is a recipe worth stealing for other low-data adaptation problems.