ProxyPose: Turning Pose Tracking into a Video Translation Problem

Paper: 2607.06555 Authors: Ruihang Zhang, Felix Taubner, Pooja Ravi, Kiriakos N. Kutulakos, David B. Lindell Categories: cs.CV

The Gap

Six-degree-of-freedom pose tracking — figuring out how an object moves and rotates in 3D from a plain video — has been around for decades. But every serious method demands something extra beyond the video itself. Traditional feature-based trackers (think SIFT or DSO-style pipelines) need textured surfaces and often a pre-built 3D model. Learning-based approaches like BundleSDF or MegaPose require object masks or depth maps as input. Foundation-model-based methods like FoundationPose still need a 3D object mesh. And all of them crumble when the surface is textureless, transparent, reflective, or deformable — precisely the cases that matter in robotics, AR, and real-world manipulation.

The gap is clean: no existing method can track 6-DoF pose of arbitrary surfaces from a monocular video alone, without any auxiliary input, on challenging materials. ProxyPose fills this by asking: what if we stop trying to “solve” pose estimation directly, and instead translate the hard problem into one we already know how to solve?

                     Problem
                        |
          What is the surface doing in 3D?
                        |
        v               v               v
   Need 3D model    Need masks     Need depth
   (MegaPose)      (BundleSDF)    (most methods)
        |               |               |
        v               v               v
   FAIL on: textureless, transparent,
   reflective, deformable surfaces
                        |
          Assumption broken: we cannot
          get reliable geometric cues
          from raw pixels alone
                        |
            Solution: shift the burden
                        |
          v                             v
   ProxyPose says:                 Instead of solving
   "don't extract geometry,        pose from pixels,
    translate pixels to pixels"     solve pose from
                                    a known proxy
                        |
            Method: video diffusion model
            translates input video into
            proxy video of colored polyhedron
                        |
            Evidence: state-of-the-art accuracy
            on standard benchmarks, works on
            faces, cameras, in-the-wild scenes
                        |
            Conclusion: the hard part of pose
            tracking can be absorbed into a
            learned translation step

The Increment

One sentence: Before this paper, 6-DoF pose tracking always required auxiliary inputs (3D models, masks, depth); after this paper, a single marked pixel in the first frame plus a video diffusion model is sufficient — and it handles the hardest materials without seeing real training data.

Core Mechanism

ProxyPose has three stages, and the elegance is in how it splits responsibilities.

Stage 1 — Input conditioning. The user provides a monocular video and marks a single pixel on the surface region of interest in the first frame. That’s it. No 3D model, no mask, no depth. This single pixel tells the system “track this spot.”

Stage 2 — Video-to-video translation. A video diffusion model (fine-tuned on synthetic data only) takes the input video and the marked pixel, and generates a *proxy video. The proxy video shows a colored polyhedron — a simple 3D shape with known geometry and known appearance — undergoing the same local rigid-body motion as the marked surface region. The diffusion model has learned, through large-scale pre-training, how to map arbitrary visual appearances (glossy metal, translucent glass, wrinkled fabric) onto this clean proxy representation. The proxy video is a “translation” of the hard problem into an easy one.

Stage 3 — Classical pose estimation. Because the proxy polyhedron’s 3D model is known by construction (its vertices, edges, and colored faces are all predetermined), recovering the 6-DoF pose at each frame is a textbook problem. Off-the-shelf Perspective-n-Point (PnP) solvers or similar classical algorithms extract the full trajectory. No learning needed here — just geometry.

 Input Video          Diffusion Model           Proxy Video
 + marked pixel           (fine-tuned)        (colored polyhedron)
       |                      |                       |
       |    condition +       |   learned              |
       +------->--------------+   translation ------->-+
                                        |
                                        v
                               Known 3D geometry
                               + known appearance
                                        |
                                        v
                               PnP solver (classical)
                                        |
                                        v
                               6-DoF pose trajectory

Here’s a structural metaphor that makes this click. Imagine you’re a detective trying to figure out the exact path of a car from blurry surveillance footage. The car is silver, half-occluded, shot through rain-smeared glass — nightmare conditions. Traditional methods say: “enhance the image, find features, match to a 3D car model.” That fails because the features are gone.

ProxyPose says: instead of analyzing the hard footage directly, hire an artist who has watched millions of hours of footage and can draw what they see — but in their own style. The artist draws the same scene, but replaces the silver car with a brightly colored toy block that has exactly the same motion. Now you, the detective, can measure the toy block’s trajectory with a ruler and a protractor (PnP solver), because you know exactly what a toy block looks like from every angle.

The video diffusion model is the artist. The polyhedron proxy is the toy block. The PnP solver is the ruler. The artist absorbs all the difficulty (rain, blur, transparency) into their translation step. The detective (pose solver) only sees clean, well-defined geometry.

Key Concepts

• Video-to-video translation as problem reformulation: The deepest idea here isn’t a new architecture — it’s a reframing. Instead of building a monolithic system that goes from raw pixels to 6-DoF pose (which is the output of every prior method), ProxyPose inserts a middle representation. Think of it like translating a French novel into Spanish before reading it, because you read Spanish fluently but your French is shaky. The “translation” doesn’t solve the problem — it converts it into a form where existing tools (classical pose solvers) work perfectly. The insight is that video diffusion models, trained on internet-scale data, have internalized enough physics and appearance understanding to perform this translation reliably.

• Proxy object: The “proxy” is a colored polyhedron — imagine a small soccer ball-like shape with distinctly colored triangular faces. Its geometry (exact shape, dimensions) is fixed and known in advance. Its appearance (which face is which color) is also known. This is critical: because the proxy’s 3D model is trivial, you never need to estimate it. Every frame of the proxy video gives you enough correspondences (colored vertices, edges) to solve for pose. It’s a universal adapter — no matter what the original surface looks like, the output is always the same well-behaved geometric object.

• Synthetic-only fine-tuning: The video diffusion model was fine-tuned using only synthetic (computer-rendered) training data — not real-world captures. This is surprising and important. It means you don’t need expensive motion-capture rigs or hand-labeled pose annotations from real scenes. The pre-trained model’s general visual knowledge transfers from synthetic to real, and the fine-tuning only teaches it the specific translation task (real appearance → proxy appearance). This dramatically lowers the data barrier.

Framework Shift

Before (mainstream approach):
                                        Known 3D model
                                              |
 Monocular Video + aux inputs -------> Direct pose
 (depth/masks/mesh)                    estimation network
                                              |
                                              v
                                        6-DoF pose

 The method must simultaneously understand appearance,
 handle materials, AND solve geometry.


After (this paper):

 Monocular Video -------> Video Diffusion -------> Proxy Video
 + one marked pixel       (translation)           (colored polyhedron)
                                                         |
   hard problem          absorbs material          Known geometry
   stays here            complexity                   |
                                                         v
                                                    PnP solver
                                                         |
                                                         v
                                                    6-DoF pose

 The hard part (materials/appearance) and the easy part
 (geometry) are cleanly separated.

From monolithic pose estimation to modular translation-then-solve, the core shift is decomposing a hard problem into a learned translation step and a trivial classical step.

Expert Assessment

Problem choice: This is a genuine, well-identified gap. The community has been throwing increasingly powerful networks at pose tracking while accumulating more and more input requirements (meshes, masks, depth). The paper correctly identifies that the real bottleneck is *material-agnostic appearance understanding, not geometric reasoning, and finds a clean way to offload that bottleneck to a pre-trained foundation model. It sits at a natural intersection of diffusion models and 3D vision — a spot that will only grow.

Method maturity: The insight is clever, not brute force. The decomposition into translation + classical solve is elegant and likely more robust than end-to-end approaches for the challenging material cases. However, I have concerns about latency — diffusion models are slow, and pose tracking often needs real-time performance. The paper doesn’t emphasize this limitation enough. There may also be simpler intermediate representations (optical flow, feature point tracks) that could partially fill the same role without a full diffusion model, and the paper doesn’t deeply explore why those alternatives fall short.

Experimental integrity: The baselines are reasonable — they compare against methods that actually use *more input information, which makes ProxyPose’s results even more impressive. Showing state-of-the-art accuracy while requiring strictly fewer inputs is a strong empirical story. The synthetic-only training claim is compelling if true, but I’d want to verify there’s no implicit data leakage between training and evaluation scenes. The extensions to face tracking and camera pose estimation are nice but feel more like demos than rigorous benchmarks — the numbers there are thin.

Writing quality: The paper reads well and the project page is polished. The main weakness is in the limitations discussion: the authors undersell the computational cost of running a video diffusion model, and they don’t clearly characterize failure modes. When does the translation break down? What happens if the marked pixel drifts off the object? A candid “here’s where it fails” section would significantly strengthen the paper. The related work section also feels like it could be tighter — there’s a long list of methods that could be grouped more thematically.

Verdict: weak accept — The core idea (pose tracking as video translation) is genuinely novel and opens a new design space, but the paper would benefit from deeper analysis of failure modes, computational cost, and ablation of the diffusion model versus lighter alternatives.

Takeaways

The most transferable idea from this paper is the “translate-then-solve” decomposition pattern. Whenever you have a problem where (a) the input domain is messy and varied, (b) the output domain is well-structured and geometrically clean, and (c) there exists a classical solver for the clean version — you can insert a learned translator between input and solver. The translator handles all the messy appearance variation; the solver handles the precise geometry. This pattern could apply to: motion estimation from satellite imagery, structural analysis from photos, or even medical imaging where you translate noisy scans into clean geometric models before measuring.

A second, more practical takeaway: video diffusion models are underutilized as general-purpose visual translators, not just generators. Most people think of diffusion models as tools for creating images. ProxyPose shows they can serve as learned “adapters” that convert one visual domain into another while preserving motion and structure. If you’re working on any vision task where the bottleneck is appearance variation rather than geometric reasoning, a fine-tuned diffusion model might be the missing translator.

A final honest note: if you need real-time pose tracking (robotics, VR), this method is not ready yet. But the idea will survive faster diffusion architectures.