PhysiFormer: Forecasting 3D Object Motion via Diffusion in World Space

Paper: 2606.27364
Authors: Yiming Chen, Yushi Lan, Andrea Vedaldi
Categories: cs.CV

The Gap

Existing approaches to learning physics from video either operate in pixel space (view-dependent, costly), or hack in latent features and enforce rigid-body or causal constraints by hand.
These neural physics models (e.g., GNN-based simulators, deep Lagrangian networks) often require ad-hoc graph structures, object-level encodings, or explicit material parameters.
They struggle with elasticity, mixed materials, and uncertainty in future states.

What’s missing is a unified, probabilistic framework that predicts 3D mesh motion without handcrafted inductive biases, works with arbitrary geometries, and can represent multiple plausible futures.

[Problem: physics is hard to learn in pixel space or latent space]
    |
    \> Assumption: direct world-coordinate vertex prediction with diffusion removes need for rigid/causal constraints
    |
    \> Method: PhysiFormer -- factorized attention over time, space, objects; denoising diffusion on vertex trajectories
    |
    \> Evidence: outperforms auto-regressive baselines; works with mixed materials, unseen shapes, more objects
    |
    \> Conclusion: coordinate-space diffusion is a viable path to general-purpose world modelling

The Increment

One sentence: Before PhysiFormer, neural physics for 3D meshes either required explicit structure (rigidity, causality) or operated in view-dependent latent spaces; after PhysiFormer, a single diffusion transformer trained on vertex coordinates can generate diverse, physically plausible motions for rigid and elastic objects without any such handcrafted biases.

Core Mechanism

At its heart, PhysiFormer is a diffusion transformer that treats future vertex positions as a random variable conditioned on the initial state (positions, velocities) and a material type tag.
During training, the model learns to reverse a diffusion process: from clean vertex trajectory data, noise is gradually added; the model is trained to predict the noise (or equivalently, the clean trajectory) given the noisy input.

During inference, the model starts from pure Gaussian noise and iteratively denoises towards a plausible future trajectory.
The transformer architecture uses factorized attention: separate attention heads for temporal, spatial (per-object), and inter-object interactions.
This keeps computation linear in the number of time steps and objects, and permutation-invariant across objects — no need to assign IDs.

The input is a sequence of vertex positions and velocities at time 0 (and optionally at past frames for velocity), plus a one-hot material vector.
The output is a sequence of future vertex positions over a fixed horizon.

[Input]          [PhysiFormer]          [Output]
  initial vertices + material type   |   future vertex trajectories
  + initial velocities               |   (multiple plausible samples)
       |                             |
       v                             v
+-----------------+        +---------------------+
| Flat embed      | ----\ | Diffusion Transformer|
| (vertex pos+vel)|      \| (factorized attn)   |
| + material one-hot   |      +---------------------+
+-----------------+             |                 |
       |                        |                 |
       +------------------------+ denoise loop    |
                                | (T steps)       |
                                v                 v
                           final vertex positions

Now, let’s map this onto a structural metaphor: think of a clay-modelling stop-motion animation.

• The clay model is the initial 3D mesh (vertex positions and velocities).
• The material tag (rigid or elastic) tells the animator whether the clay is hard (can only translate/rotate) or soft (can stretch and bend).
• The diffusion process is like the animator working backwards from a rough blob to the final frame: at the start (high noise) it’s a shapeless mess; after each denoising step the animator adds more detail, refining the mesh until a clean, physically-plausible pose emerges. Running the process multiple times yields different “takes” of the same scene (different plausible motions).
• The factorized attention is like the animator’s divided workspace: one viewfinder tracks time (checking frames before and after), another viewfinder tracks each clay character’s own body (spatial structure), and a final viewfinder looks at interactions between multiple clay figures. This modular attention lets him handle many characters without getting tangled.

So PhysiFormer learns, without being told any physics rules, how to “animate” 3D meshes by practicing on 100k recorded trajectories (the training data). The result: it can handle new shapes, new mixtures of rigid and elastic objects, and even simulate collisions — all by denoising in world space.

Key Concepts

• Diffusion for trajectory prediction:
  Instead of predicting one deterministic trajectory (like autoregressive models), diffusion generates a distribution over trajectories.
  Intuition: imagine you want to guess where a bouncing ball will be in 3 seconds. If you only know initial position and velocity, there is uncertainty due to spin, air, etc.
  Diffusion gives you many plausible paths. At each denoising step, the model corrects the guess a little based on learned physics, starting from total randomness.
  Example: three initial conditions may lead to 5 different bouncing patterns; diffusion can sample all of them.

• Factorized attention over time, space, objects:
  Standard self-attention over a long sequence of all vertices across all time steps would be O(N^2) huge.
  PhysiFormer splits the attention into three dimensions:

  • Time: each vertex attends to its own past/future positions (keeping motion smooth).
  • Space: each vertex attends to other vertices in the same object at the same time (preserving shape).
  • Objects: the model attends between objects globally (handling interactions).
    This keeps computation linear in each dimension, and is permutation-invariant across objects (no object ID needed).
    Intuition: like in a group meeting where you talk about the agenda (time), look around at your own team (space), and occasionally pivot to another team (objects) for coordination.

• World coordinate representation:
  Instead of predicting pixel locations (which depend on camera angle) or latent codes, the model directly outputs 3D coordinates of mesh vertices in a fixed world frame.
  This is view-invariant: the same physical scene can be rendered from any angle.
  It also means the model doesn’t need to learn to disentangle camera motion from object motion; it’s learning true 3D physics.

Framework Shift

Before (mainstream approach): neural physics in latent/graph space

[Video frames] -> [Encoder to latent graph] -> [Graph NN + physics bias] -> [Decoder to mesh]
   (+ camera pose)          (rigid latent)           (hardcoded constraints)     (pixel projection)

After (this paper):

[Initial mesh + velocity + material] -> [Diffusion Transformer (factorized attn)] -> [Future vertex positions (world coords)]
   (world coordinates)                                      (no explicit constraints)  (multiple samples)

One sentence: From pixel/latent space with handcrafted physics biases to direct world-coordinate diffusion without inductive biases.

Expert Assessment

Problem choice: Real and well-motivated. Simulating physics for 3D meshes is crucial for robotics and graphics; existing neural methods are ad-hoc. The gap — removing explicit constraints while handling uncertainty — is genuine.

Method maturity: The ingredients (diffusion, transformer, factorized attention) are all known, but the combination and the novel choice of vertex-level diffusion in world space is a clear advance. It’s clever, not brute-force. Could simpler models (e.g., MLP+momentum) work for simple rigid cases? Possibly, but the paper’s contribution is handling complexity (elastic, multi-object) without special casing.

Experimental integrity: Baselines are fair: they compare against autoregressive transformer and GNN variants. The metrics (trajectory error, rigidity preservation, momentum consistency) are appropriate. One red flag: they train on 100k simulated trajectories; real-world generalization (e.g., real physics) is not tested. But for a CV paper, the numbers hold up.

Writing quality: The abstract and intro are clear. Section 3 (method) could be heavier to read. I’d rewrite the attention factorisation explanation with a cleaner diagram and an example. The paper would benefit from a figure showing the denoising process on a specific elastic object.

Verdict: weak accept — The core idea is solid and the results are strong, but the evaluation is limited to synthetic data and some technical details are under-explained.

Takeaways

• Practitioners can steal the factorized attention design for any problem involving sequences of 3D points (e.g., mesh deformation, point cloud dynamics). It’s a cheap way to handle space, time, and objects simultaneously.
• The world-coordinate diffusion idea is transferable to other prediction tasks where you want uncertainty and view-invariance (e.g., human pose over time, particle trajectory forecasting).
• Tip: They used a simple velocity calculation from two initial frames. If you’re building a physical simulator, precomputing velocity from the first two frames is a cheap way to give the model initial dynamics.