FaceCam: Teaching Video Models to Move the Camera Around Faces Without Breaking Them

Paper: 2603.05506 Authors: Weijie Lyu, Ming-Hsuan Yang, Zhixin Shu Categories: cs.CV

The Gap

Video generation models like Runway and Pika can now take camera control inputs — you tell them “dolly forward” or “pan left” and they try to comply. But when you apply these to portrait videos, faces warp, stretch, and break. The problem isn’t the generation quality per se; it’s that existing camera representations are scale-ambiguous. A “move forward 2 meters” instruction means nothing without knowing how far away the subject is. Prior work either uses generic 3D camera parameters (which don’t account for face-specific geometry) or reconstructs 3D meshes first (which introduces errors that propagate through generation). Both approaches treat portraits like any other scene, ignoring that faces have predictable structure we can exploit.

Problem: Camera control on portraits creates distortions
   |
   v
Assumption: Face geometry is predictable, unlike generic scenes
   |
   v
Method: Scale-aware conditioning using face landmarks (no 3D reconstruction)
   |
   v
Evidence: Ava-256 + in-the-wild videos show better geometry preservation
   |
   v
Conclusion: Domain-specific representations beat generic 3D priors for faces

The Increment

One sentence: Before FaceCam, controlling camera motion in portrait video generation meant choosing between scale-ambiguous parameters or error-prone 3D reconstruction; after, you get deterministic control using 2D face landmarks that inherently encode scale.

Core Mechanism

FaceCam has three components working in sequence. First, a scale-aware camera encoder takes your desired camera trajectory and converts it into a representation grounded in face geometry — specifically, it uses 2D facial landmarks to define camera motion relative to the face itself, not abstract 3D space. This sidesteps the “how far is 2 meters?” problem entirely because motion is expressed as changes in landmark positions and scales.

Second, a video generation backbone (built on diffusion models) takes your input portrait video and the encoded camera trajectory, then generates new frames. The key is that camera conditioning happens at multiple scales through cross-attention layers, so both coarse motion (head turning) and fine details (eye gaze) respect the camera instruction.

Third, training data comes from two sources: multi-view studio captures (where you have ground truth camera positions) and in-the-wild monocular videos (where you synthesize camera motion or stitch multiple shots). The synthetic motion strategy generates plausible camera trajectories by interpolating between static frames. The multi-shot stitching takes videos with cuts and creates smooth camera transitions between them, effectively teaching the model continuous motion from discontinuous data.

Input Portrait Video
        |
        v
   [Face Landmark]  <--- Desired Camera Trajectory
    Detection              |
        |                  v
        |            [Scale-Aware]
        |             Encoder
        |                  |
        +------------------+
                 |
                 v
          [Video Diffusion]
           Model (U-Net)
                 |
                 v
          Output Video
       (new camera view)

Think of FaceCam like a puppeteer controlling a marionette. Traditional camera control is like shouting directions from the audience — “move left!” — but the puppeteer doesn’t know if you mean stage left or their left, or how far. FaceCam instead attaches strings directly to the puppet’s joints (face landmarks). When you want the camera to move closer, you pull the strings to make the puppet’s face appear larger in frame. The strings (landmark positions) encode both direction and scale simultaneously. The puppeteer (video model) just follows the string tension, no guessing required. The training regime is like practicing with both a professional puppet (studio multi-view data) and a hand-me-down puppet with some strings missing (in-the-wild monocular data), so the puppeteer learns to work with whatever strings are available.

Key Concepts

• Scale-Aware Conditioning: Imagine you’re directing a cinematographer to frame a portrait shot. If you say “move the camera 1 meter forward,” they’ll ask “from where?” But if you say “frame the face so the eyes are 30% of frame width,” that’s unambiguous. Scale-aware conditioning does the latter — it describes camera position relative to observable face features (landmark distances, face bounding box size) rather than abstract 3D coordinates. A close-up has landmarks spread wide across the frame; a distant shot has them clustered small. The model learns this mapping, so “make landmarks 2x bigger” deterministically means “move camera closer by a specific amount” without ever computing depth.

• Synthetic Camera Motion: You have a static video of someone talking. How do you teach a model to move the camera around them? Synthetic motion generates plausible trajectories by treating the static video as a keyframe and interpolating camera positions between it and hypothetical other views. It’s like having a photo of a building and imagining what it looks like from 10 degrees to the left — you don’t have that photo, but you can infer constraints (windows stay rectangular, walls stay parallel). For faces, the constraints are stronger: eyes stay symmetric, nose stays centered. The model learns these invariants and generates intermediate frames that respect them.

• Multi-Shot Stitching: Movie scenes often have cuts — close-up, then wide shot, then over-the-shoulder. Each shot is static, but the sequence implies camera motion. Multi-shot stitching takes these discontinuous clips and trains the model to generate smooth transitions between them. It’s teaching the model to “fill in” the camera movement that would connect shot A to shot B. The clever bit: this turns abundant static footage into training data for continuous camera control, without needing actual moving cameras during capture.

Framework Shift

Before (mainstream approach):        After (this paper):

Input Video                          Input Video
     |                                    |
     v                                    v
[3D Reconstruction]                 [2D Face Landmarks]
  (mesh/depth)                        (direct observation)
     |                                    |
     | (errors propagate)                 | (no reconstruction)
     v                                    v
[Generic Camera Params]             [Scale-Aware Encoding]
  (x,y,z,roll,pitch,yaw)              (landmark deltas + scale)
     |                                    |
     v                                    v
[Video Generation]                  [Video Generation]
  (distortions)                       (geometry preserved)

From “reconstruct 3D then apply generic camera math” to “use 2D face structure as the camera reference frame itself,” the core shift is eliminating the reconstruction bottleneck by making the conditioning domain-specific.

Expert Assessment

Problem choice: This is a real gap, not manufactured. Portrait video generation is commercially valuable (content creation, virtual production), and camera control is the obvious next frontier after basic generation quality. The failure mode — geometric distortions on faces — is highly visible and breaks immersion. The problem sits at the intersection of video generation (hot) and 3D-aware synthesis (established), making it timely.

Method maturity: The scale-aware representation is genuinely clever — it’s not obvious that 2D landmarks contain enough information to replace 3D camera parameters, but faces are constrained enough that it works. However, the training data strategies (synthetic motion, multi-shot stitching) feel like workarounds for not having real moving-camera portrait datasets. They’re practical, but I’d want to see ablations showing how much performance degrades compared to true multi-view training data.

Experimental integrity: Baselines are reasonable (CameraCtrl, MotionCtrl), though I’d like to see comparisons against 3D-aware methods like EG3D adapted for video. The Ava-256 dataset is standard, and in-the-wild results are convincing. One red flag: no user study quantifying “geometric distortion” — the paper relies on visual inspection and identity preservation metrics, which are proxies. Distortion is subjective; you need human raters.

Writing quality: The abstract and intro are crisp, but the method section buries the key insight (why landmarks encode scale) under implementation details. Figure 2 should come earlier. The related work section is thorough but reads like a literature review, not a positioning statement. Rewriting it to emphasize “why existing camera representations fail for faces” would sharpen the narrative.

Verdict: weak accept — solves a real problem with a clever domain-specific insight, but the evaluation could be more rigorous and the writing could better highlight the core contribution.

Takeaways

The big transferable idea: when your domain has strong structural priors (faces, hands, rigid objects), don’t force it through generic 3D representations. Design conditioning signals that exploit domain structure directly. For faces, 2D landmarks work because facial geometry is constrained. For hands, bone lengths are fixed. For cars, wheels stay circular. The pattern is: find the invariants, encode them in your conditioning, skip the reconstruction step.

Concretely, practitioners working on controllable generation for specific object categories should ask: “What 2D observations uniquely determine 3D configuration in my domain?” For faces, it’s landmark positions and scales. For your domain, it might be silhouette curvature, part symmetries, or texture gradients. Build your conditioning around those, not generic camera matrices.

The training data strategies are also worth stealing. If you have static multi-view captures, synthetic motion generation lets you create continuous trajectories. If you have monocular videos with cuts, multi-shot stitching turns them into motion training data. Both are ways to bootstrap dynamic control from static or discontinuous data.