TryOnCrafter: Camera-Controllable Video Virtual Try-on with a 4D Proxy

Paper: 2606.26092
Authors: Hao Sun, Hao Yan, Mengting Chen, Quanjian Song, Yu Li, Juan Cao, Jinsong Lan, Xiaoyong Zhu, Bo Zheng, Sheng Tang
Categories: cs.CV

The Gap

Existing Video Virtual Try-on (VVT) methods (e.g., MagicAnimate, AnimateAnyone, OOTDiffusion-based pipelines) synthesize a garment overlay on a person in a video by conditioning on the source video frames.
They assume the camera trajectory is fixed—i.e., they passively follow the viewpoint of the input video.
If you want to orbit around the person, zoom in, or create a “bullet time” freeze-frame, the model has no mechanism to accommodate it.
The field had not even defined this as a separate problem.

This paper identifies the limitation explicitly and coins the term Camera-controllable Video Virtual Try-on (CaM-VVT).
The core gap: existing VVT cannot decouple garment appearance from camera viewpoint, so any novel view induces structural collapse or texture flickering.

Logic topology:

[Existing VVT pipelines]  
   |  
   +--> Conditioned on source video only  
   +--> No way to specify target camera path  
   |  
   v  
Gap: viewpoint = stuck in source trajectory  
   |  
   v  
[This paper: CaM-VVT]  
   |  
   +--> Define task: generate video under arbitrary camera  
   +--> Need: viewpoint-agnostic texture + motion consistency  
   |  
   v  
[Method]  [Evidence]  [Conclusion]  
   |           |           |  
   v           v           v  
Decouple human from environment via a Renderable 4D Proxy  
   |  
   +--> 2D try-on -> 3DGS avatar -> SMPL-X animation + BG point cloud  
   +--> Proxy-Anchored Video DiT uses proxy renderings as geometric anchor  

The Increment

One sentence: Before TryOnCrafter, VVT was a 2D inpainting problem with fixed viewpoint; after, it becomes a 4D conditional generation problem where you can freely move the camera.

Core Mechanism

The method has two major blocks: (1) building a Renderable 4D Try-on Proxy, and (2) using that proxy to anchor a Video Diffusion Transformer (DiT).

First, a 2D try-on image is generated from the person’s original appearance and the target garment using an off-the-shelf inpainting model (e.g., OOTDiffusion).
This 2D prior is lifted to 3D Gaussian Splatting (3DGS) — a cloud of colored, opaque Gaussians that represent the clothed human.
The SMPL-X body model is fitted to the source video to extract a pose sequence.
The 3DGS avatar is then animated via Linear Blend Skinning (LBS) so that the Gaussians move according to the skeleton, producing a temporally consistent 3D representation of the wearing person.
Separately, a static background point cloud is reconstructed from the original video.
The animated 3DGS avatar and background point cloud are metrically aligned and form the 4D proxy: a lightweight, renderable representation that exists for every frame and can be viewed from any camera.

Second, the Proxy-Anchored Video DiT takes as input:

• RGB renderings of the 4D proxy from the target camera trajectory (low-quality but structurally correct),
• a noisy video latent,
• and optional text / garment reference.

The DiT learns to refine the proxy renderings into a photorealistic video, adding fine texture, shadows, and detail while preserving the geometric and motion structure provided by the proxy.
The proxy acts as a “geometric anchor” so the model never has to hallucinate the shape of the person or the background layout.

[Person video] [Garment image] [Target camera traj]  
      |              |                  |  
      v              v                  v  
[2D try-on prior]-->[3DGS avatar]  
                         |  
                    [SMPL-X seq]  
                         |  
                    [Animated 3DGS via LBS]  
                         |  
                    +----+----+  
                    |         |  
              [BG point cloud] |  
                    |         |  
                    v         v  
              +------+---------+  
              | 4D Try-on Proxy |  (renderable from any camera)  
              +-----------------+  
                    |  
                    v (render target camera)  
              [Proxy renderings]  
                    |  
                    v  
              +------+--------+  
              | Proxy-Anchored |  
              | Video DiT      |  
              +------+--------+  
                    |  
                    v  
              [Photorealistic video]  

Structural metaphor: Think of the 4D proxy as a marionette with a garment stitched on.

• The 2D try-on is the tailor’s sketch;
• 3DGS is the puppet’s body (made of malleable light‑emitting beads);
• SMPL-X is the puppeteer’s rods that dictate joint angles;
• LBS moves the beads according to the rods;
• The background point cloud is the stage set.

The Proxy-Anchored DiT is the master painter who watches the puppet from any seat in the audience (camera trajectory) and paints the final detailed performance on top of the moving beads.
The puppet guarantees anatomy; the painter only adds realistic shading, fabric texture, and ambient effects.
If the puppet’s arm is in the right place, the painter cannot make it look wrong — he can only make it look real.

Key Concepts

• 3D Gaussian Splatting (3DGS): A scene representation as a collection of 3D Gaussian ellipsoids with color, opacity, and shape parameters.
  It renders fast via differentiable splatting into a 2D image.
  Unlike mesh or voxels, it is easy to deform — you can move each Gaussian’s center using bone weights (LBS).
  In this paper, 3DGS is used to represent the clothed human so that the garment can “follow” the skeleton without topological constraints.

• SMPL-X and Linear Blend Skinning (LBS): SMPL-X is a parametric human model that gives a pose vector (joint angles) and shape vector.
  LBS takes vertex positions (or Gaussian centers) and a skinning weight per vertex per bone, then rotates and translates each vertex according to the bone transformations.
  The paper uses SMPL-X to pose the 3DGS avatar for each frame of the source video, ensuring the garment moves naturally with the body.

• Proxy-Anchored Video Diffusion: Instead of letting a diffusion model guess the 3D structure of a scene from 2D conditioning (like depth maps), this method provides a full 4D geometric prior (the proxy) that evolves over time.
  The diffusion model only has to “beautify” the proxy renderings — a much simpler task.
  This is analogous to inpainting a texture onto a pre‑rendered geometry, but done per frame with temporal coherence.

Framework Shift

Napkin sketch comparing the old paradigm vs. this paper’s paradigm:

Before (mainstream VVT):  
[Source video] -> [2D try-on network] -> [Output video]  
      |                 |                      |  
   camera fixed   camera unchanged       camera same as source  
   garment from   replaces garment       no viewpoint control  
   source         region in 2D  

After (TryOnCrafter):  
[Source video] + [Garment] + [Target camera trajectory]  
      |                 |                |  
      v                 v                v  
[Renderable 4D Proxy] (3DGS + BG PC)     |  
      |                                   |  
      v                                   v  
[Proxy renderings @ target cam] --> [Video DiT] --> [Output video]  
                                          |               |  
                                       geometric      arbitrary camera,  
                                       anchor         consistent motion  

One sentence: From implicit 2D inpainting with a frozen camera to explicit 4D‑anchored generation with controllable camera — the core shift is decoupling the garment appearance from the viewpoint.

Expert Assessment

Problem choice: Real gap.
The paper correctly identifies a blind spot in VVT research: every prior work treats the camera as an uncontrollable given.
Practically, e‑commerce and virtual try‑on apps desperately want 360° viewing.
Academically, it merges VVT with novel view synthesis, which is a natural progression.

Method maturity: Clever insight, moderately engineered.
The decoupling into a proxy that handles structure, then a diffusion model that adds detail is a clean divide‑and‑conquer.
However, the proxy construction depends on several off‑the‑shelf models (2D try‑on, depth estimation, SMPL‑X fitting, 3DGS lifting from a single view) — each is a potential failure point.
Could there be a simpler end‑to‑end approach using latent 3D features? Possibly, but the authors’ explicit proxy can be debugged and edited, which is a practical advantage.

Experimental integrity: Fair.
Baselines include MagicAnimate and AnimateAnyone adapted to receive camera parameters (as far as they allow).
Ablation studies show that removing the proxy or replacing it with a weaker condition (e.g., depth maps) degrades consistency and quality.
Numbers (FID, CLIP score, user study) are solid.
One red flag: the 2D try‑on prior (OOTDiffusion) is used as a black box — its failures (e.g., wrong garment shape) propagate.
The paper acknowledges this but does not quantify how often it succeeds.

Writing quality: Dense but clear.
The introduction could better motivate the “bullet time” and “reloc” applications earlier.
The method section would benefit from a figure showing the proxy construction pipeline (words alone are heavy).
The best technical section is 3.3 (Proxy‑Anchored DiT) — it explains the conditioning mechanism precisely.

Verdict: Weak accept — solves a real, previously undefined problem with a plausible method.
The impact depends on how robust the 3DGS proxy is when transferred to real‑world videos with occlusions or complex background.
Nonetheless, it opens a clear future direction.

Takeaways

• Renderable 4D proxy as an intermediate representation is a transferable idea: for any video generation task that requires structural control (e.g., rendering a character performing a specific action from any angle), you can build a lightweight 3D proxy and let a diffusion model add realism.
• 3DGS + LBS is a practical recipe for animatable clothed‑human representation from a single video; it could be used for fast avatar re‑targeting in other contexts (e.g., digital humans, gaming).
• Proxy‑anchored conditioning — providing the diffusion model with a geometrically correct but low‑quality video as anchor — is a general strategy that reduces the generation burden to texture and lighting synthesis.
  This could be applied to any domain where a cheap renderer exists (architecture, automotive, etc.).