Flex4DHuman: Flexible Multi-view Video Diffusion for 4D Human Reconstruction

Paper: 2606.13655 Authors: Jen-Hao Cheng, Yipeng Wang, Hao Zhang, Gengshan Yang, Jenq-Neng Hwang Categories: cs.CV, cs.GR

The Gap

Existing 4D human reconstruction methods fall into two camps. The first relies on multi-view camera rigs synchronized in time (e.g., DNA-Rendering, ActorsHQ) – expensive, static setups that don’t scale. The second works from monocular video but depends on explicit geometry priors: SMPL body models, depth maps, normal maps, or skeleton poses. These priors are brittle under occlusion, loose clothing, animal subjects, or rapid motion; they also require separate pre-trained estimators that accumulate errors.

Flex4DHuman removes these priors entirely. It asks: can we directly generate multi-view videos from a single moving camera without ever recovering a skeleton or depth map? The answer is yes, using a video diffusion model conditioned only on relative camera poses. The logical chain:

[Problem: 4D from monocular video -- heavy priors, poor generalization]
        |
        v
[Assumption: No geometry priors needed -- just relative camera poses]
        |
        v
[Method: Multi-view video diffusion with 5-axis positional encoding]
        |
        v
[Evidence: SOTA on DNA-Rendering & ActorsHQ + zero-shot to animals]
        |
        v
[Conclusion: Practical scalable 4D from casual monocular videos]

The Increment

One sentence: Before Flex4DHuman, 4D human reconstruction from monocular video required explicit geometry priors (SMPL, depth, normals) and multi-view capture setups; after Flex4DHuman, you can generate dense multi-view videos directly from a single static-camera video using only relative camera pose conditioning, then lift them to 4D Gaussians.

Core Mechanism

Flex4DHuman is built on Wan 2.1 1.3B, a pretrained text-to-video diffusion model. The backbone is kept, but the positional encoding is extended: a standard spatio-temporal RoPE covers the (x, y) spatial grid and the time step; Flex4DHuman adds two extra axes – a view index (which camera viewpoint does this frame belong to?) and a continuous relative camera pose (SE(3) transformation from reference to target view). This five-axis positional encoding is the only way camera information enters the model; no explicit geometry is injected.

The training follows a three-stage curriculum: (1) pose following – the model learns to generate a single target view conditioned on a reference view and their relative pose; (2) flexible reference-to-target view generation – it learns to handle arbitrary reference-to-target pairs; (3) temporal rollout – it learns to generate a full multi-view video sequence by attending to clean historical target-view tokens from earlier time steps. Additionally, multi-view captions are added during training to enable text-controlled generation at test time (e.g., “man in red shirt doing jumping jacks”).

Once multi-view videos are generated, an off-the-shelf 4D Gaussian Splatting pipeline (similar to 4DGS) fits 4D Gaussians to the rendered frames, producing a dynamic 3D representation.

[Input: Monocular video frames + target camera poses (relative)]
        |
        v
+-------------------------------------------+
| Wan 2.1 1.3B backbone                     |
| 5-axis positional encoding:               |
|   (x, y, t, view_idx, SE(3) rel_pose)    |
| No geometry priors used                   |
+-------------------------------------------+
        |
        v
[Output: Dense multi-view video frames (N viewpoints x T frames)]
        |
        v
[4D Gaussian Splatting] -> Dynamic 4D Gaussians (representation)

Let me give you a load-bearing metaphor. Think of Flex4DHuman as a film director who only watched one camera’s footage but needs to direct a multi-camera shoot. Normal directors use body doubles, rigging plans, and skeleton charts (geometry priors). This director instead imagines: “If the second camera were three feet to my right and pointing left, what would it see?” He doesn’t need to know where the actor’s elbow is in 3D; he only needs the relative pose between cameras. The five-axis positional encoding is like giving each camera crew member a unique ID and a spatial relation note. The three-stage curriculum is how the assistant learns: first to mimic a single other camera, then to switch between any camera pair, then to sustain the illusion over a long scene (temporal rollout). The multi-view captions are like the script – you can shout “red shirt” and the director will shape the generation accordingly.

Key Concepts

• Relative camera pose conditioning: Instead of providing absolute world coordinates, Flex4DHuman encodes the relative SE(3) transformation from a reference view to the target view. This makes the model view-relationship-aware, not world-aware. For example, “camera B is 2m to the right of camera A, rotated 30° left.” The model learns to hallucinate what camera B sees given camera A’s image. This is far easier to generalize because it doesn’t require a consistent global coordinate system across scenes.

• 5-axis positional encoding (5D RoPE): The original spatio-temporal RoPE encodes positions in a 3D grid (x, y, t). Flex4DHuman adds two dimensions: a discrete view index (integer) and a continuous 6D relative camera pose (3D rotation + 3D translation). The view index helps the model distinguish between different output viewpoints; the continuous SE(3) encodes the geometric relationship between views. All five axes are mixed into the attention mechanism via rotary embeddings. Without these, the model would treat all target views identically, unable to enforce geometric consistency.

• Three-stage curriculum learning: Training a multi-view video diffusion model from scratch is unstable. The curriculum decomposes the problem: Stage 1 (pose following) – given one reference frame, generate one target frame with a known relative pose. Stage 2 (flexible reference-target) – randomize which view is reference vs target (so the model handles asymmetric conditioning). Stage 3 (temporal rollout) – extend to full sequences by feeding earlier generated target-view tokens (cleaned) as context for future frames. This stepwise learning avoids catastrophic forgetting and stabilizes long-horizon generation.

Framework Shift

The shift is from geometry-first to view-first.

Before (mainstream approach):        After (this paper):
Monocular video                      Monocular video
    |                                    |
    v                                    v
[Estimate geometry priors]           [Flex4DHuman diffusion]
(SMPL, depth, normal)                (no geometry priors)
    |                                    |
    v                                    v
[Back-project to 3D + render]        [Dense multi-view video]
(implicit multi-view)                (explicit generated)
    |                                    |
    v                                    v
[Refine 4D shape]                    [4D Gaussian Splatting]

One sentence: From “estimate geometry from monocular video then reconstruct 4D” to “generate virtual multi-views from monocular video then fit 4D Gaussians” – the core shift is replacing brittle geometric inference with learned multi-view video generation.

Expert Assessment

Problem choice: This is a real gap. Casual 4D capture (e.g., filming a person with a phone camera) is the holy grail for VR/AR and gaming. Prior methods either require multi-camera studios or heavy geometric priors that fail on deformable, clothed, or animal subjects. Flex4DHuman attacks the right bottleneck – removing the need for SfM/body models.

Method maturity: Clever insight – use relative camera pose as the only conditioning, and push geometry to the final Gaussian fitting stage. The reliance on a large pretrained model (Wan 2.1 1.3B) is pragmatic but also a “brute force” shortcut: the intellectual contribution is the 5-axis RoPE and curriculum training, not the base architecture. Simpler approaches like fine-tuning a smaller model on relative pose conditioning might have been explored but likely underperform.

Experimental integrity: Baselines are fair (compared against prior multi-view generation and 4D reconstruction methods on DNA-Rendering and ActorsHQ). The zero-shot animal generalization is impressive. However, all evaluation is on datasets with ground-truth multi-view; no real-world monocular video results are shown (the paper claims “monocular static-camera videos” but experiments use dataset videos that are essentially multi-view). Also, ablation of the three-stage curriculum is missing – how much does each stage contribute? A minor red flag: the 4D Gaussian Splatting stage is off-the-shelf; the reconstruction quality might depend heavily on its quality.

Writing quality: Well-structured, but the experimental section is dense and lacks clear visual comparisons of the generated multi-view frames (only tables of metrics). A figure showing generated vs. ground-truth multi-view for a challenging case (e.g., loose clothing) would dramatically improve the paper.

Verdict: weak accept – The paper makes a practical step forward in 4D content creation, but the novelty is incremental (extending RoPE + curriculum) and there’s a risk the community will replicate it with smaller, cheaper models.

Takeaways

• 5-axis positional encoding recipe: If you work on multi-view generation, you can directly borrow the idea of adding view index and relative SE(3) to RoPE. This is a generalizable trick for any model that needs to condition on camera relationships.

• Curriculum for video diffusion: The three-stage training (pose following -> flexible reference-target -> temporal rollout) is a template for breaking down complex spatio-temporal generation tasks. You can apply it to novel view synthesis from video, multi-object tracking, or even 4D scene generation.

• Offloading geometry to the end: The design philosophy of “generate views first, reconstruct geometry later” using a diffusion model is portable. Instead of trying to estimate 3D in latent space, let the video diffusion handle viewpoint diversity, then use a classical 3D fitting algorithm (here Gaussian splatting) to lift to 4D. This separation of concerns simplifies each stage.