Deform360: When Robots Learn to Feel and See Deformable Objects at Scale

Paper: 2607.05390 Authors: Hongyu Li, Wanjia Fu, Xiaoyan Cong, Zekun Li, Binghao Huang, Hanxiao Jiang, Xintong He, Yiqing Liang, Rao Fu, Tao Lu Categories: cs.RO, cs.CV

The Gap

Here’s the situation: robots are getting pretty good at manipulating rigid objects — pick up the mug, place the mug, done. But the moment you hand a robot a towel or a piece of bread, things fall apart. Deformable objects have near-infinite internal degrees of freedom. A cloth can fold, crumple, stretch, and drape in ways that are extremely hard to predict.

Two camps have emerged to tackle this. 2D video prediction models (think diffusion-based video generators) learn dynamics in pixel space — they’re scalable and capture rich appearance, but they’re fundamentally “flat”: they don’t understand 3D structure or contact physics. 3D particle-based models (like GNNs or point-cloud dynamics simulators) work in explicit 3D geometry — they’re more physically grounded, but they struggle with scalability and require expensive 3D annotations. The problem? Nobody has had a big enough, diverse enough real-world dataset to actually compare these two paradigms on equal footing. Existing datasets are either small-scale, synthetic, rigid-only, or lack tactile information. You can’t settle a debate when both sides are fighting with toy data.

Problem:
  Deformable objects are hard to predict (high-dim state, complex contact)
     |
     v
Two competing paradigms exist:
  [2D video models] <--scalable, appearance-rich, no 3D priors
  [3D particle models] <--physics-grounded, hard to scale, need 3D annotations
     |
     v
Gap: No large-scale real-world dataset to compare them fairly
     |       (existing data: small, synthetic, rigid, no tactile)
     v
Deform360: 198 objects, 1980 sequences, 215+ hrs
  41 cameras + bimanual tactile grippers
  markerless 3D tracking pipeline
     |
     v
Systematic comparison of 2D vs 3D models
  + robot planning demo on deformable objects
     |
     v
Conclusion: 2D models scale better but miss structure;
            3D models capture geometry but struggle with diversity.
            Trade-off between structural priors and scalability.

The Increment

One sentence: Before this paper, we had opinions about 2D vs 3D deformable world models; after this paper, we have data — 215 hours of it, from 198 real objects, with synchronized vision and touch.

Core Mechanism

Deform360 is primarily a dataset paper, but it’s not “just” a dataset. The real engineering challenge is the pipeline that produces the data. Let me walk through it.

Data capture hardware: A rig with 41 surround-view cameras captures the deformable object from every angle during manipulation. The robot uses bimanual tactile grippers (one in each hand), so you get both the global motion (cameras) and local contact information (tactile sensors) simultaneously. The 198 objects span daily-life categories — towels, bags, food items, clothing — chosen to cover a range of material stiffness, elasticity, and surface properties.

Markerless visuotactile 3D tracking: This is the technical contribution within the dataset pipeline. Previous visuotactile approaches often relied on fiducial markers (stickers or patterns on the object) to track deformation. Deform360’s pipeline extracts dense 3D geometry and motion without markers, using multi-view reconstruction fused with tactile signals. This means you can track deformation on arbitrary objects without prep work — crucial for scaling to 198 diverse objects.

Benchmark evaluation: With the dataset in hand, the authors run a systematic comparison. They take representative 2D video prediction models and 3D particle-based models, train them on the same data, and evaluate prediction quality across multiple dimensions: visual fidelity, geometric accuracy, contact prediction, and generalization to unseen objects.

Data Capture
  +-- 41 surround-view cameras --> multi-view RGB frames
  +-- bimanual tactile grippers --> contact force + local geometry
        |
        v
Markerless Tracking Pipeline
  multi-view RGB --> 3D reconstruction (dense geometry)
                         |
  tactile signals -------+--> fused visuotactile 3D tracking
                         |
                         v
                   per-frame dense point clouds + motion fields
        |
        v
Deform360 Dataset
  198 objects, 1980 sequences, 215+ hrs
  multi-view video + tactile + 3D geometry
        |
        v
Benchmark Evaluation
  +-- 2D video models (train + evaluate)
  +-- 3D particle models (train + evaluate)
        |
        v
Metrics: visual quality, geometric accuracy,
         contact prediction, generalization
        |
        v
Robot Planning Demo: deformable manipulation tasks

Now, a structural metaphor to make this stick. Think of Deform360 as a cooking school’s masterclass recording system.

• The 41 cameras are like having 41 students standing around the chef, each filming from a different angle — you never miss what the dough looks like from behind.
• The tactile grippers are the chef’s own hands — they feel how the dough resists, where it sticks, how much pressure is needed. Video alone can’t tell you that.
• The markerless tracking pipeline is the difference between asking the chef to put toothpicks in the dough so you can track it (markers), versus just watching the dough move naturally (markerless). The former limits what you can study; the latter lets you film anything.
• The benchmark comparison is like having two cooking schools — one teaches by showing videos (2D models), the other by building physical models of the food (3D models) — and finally giving them the same masterclass footage to learn from, then testing who produces better dishes.
• The robot planning demo is the graduation exam: can the students actually cook something new?

Without the analogy: it’s a dataset. With the analogy: it’s the masterclass recording system that lets you finally decide whether learning by watching videos beats learning by building physical replicas.

Key Concepts

• World Model for Deformable Objects: A world model is a learned simulator — you show it the current state of the world, give it an action, and it predicts what happens next. For rigid objects, this is already hard. For deformable objects, it’s nightmarish because the “state” isn’t just position and orientation — it’s the entire 3D shape of a towel that just got crumpled. Imagine trying to predict the exact shape of a piece of paper after someone crumples it, without actually seeing the crumpling. That’s the world model problem for deformables.

• Visuotactile Fusion: Vision tells you what the object looks like from the outside. Touch tells you what’s happening at the contact point — force, pressure, slip, local deformation. Neither alone is sufficient. A camera might see a bag being squeezed, but only the tactile sensor knows how much force is being applied and whether the bag is about to slip. Fusion means combining both signals into a unified representation. The analogy: vision is like reading about swimming, touch is like being in the water — you need both to learn.

• Structural Priors vs. Scalability Trade-off: A structural prior is a built-in assumption that helps a model learn faster but limits what it can represent. 3D particle models have strong structural priors (they “know” objects are made of particles in 3D space), which helps with geometric accuracy but makes them brittle when the real world doesn’t match the assumption. 2D video models have almost no structural priors (they just predict pixels), which makes them flexible and scalable but physically ungrounded. The paper’s key insight is that neither dominates — it’s a genuine trade-off, and the sweet spot probably lies somewhere in between.

Framework Shift

Before (mainstream approach):              After (this paper):
                                          
  [Small / synthetic dataset]              [Large-scale real-world dataset]
         |                                         |
         v                                         v
  Study one model type                   Systematic comparison of
  in isolation                           both paradigms on equal footing
         |                                         |
         v                                         v
  Claim: "my approach works"             Finding: trade-off is real,
  (on limited benchmarks)                neither dominates
         |                                         |
         v                                         v
  No tactile, no multi-view,             41 cameras + tactile +
  no dense 3D geometry                   dense 3D tracking

From isolated claims on toy data to systematic comparison on large-scale real data, the core shift is turning a tribal debate into a data-driven conversation.

Expert Assessment

Problem choice: This is a genuine gap. The deformable manipulation community has been hand-waving about 2D vs 3D for years, and the lack of a large-scale benchmark is a real bottleneck. The field’s trajectory clearly points toward needing exactly this kind of infrastructure. That said, this is fundamentally a dataset paper — the intellectual novelty is more in engineering and curation than in algorithmic insight.

Method maturity: The markerless tracking pipeline is solid engineering work, though the paper would benefit from more ablation on its accuracy. The benchmark evaluation is fair in scope but feels somewhat preliminary — the models compared are representatives, not exhaustive. There’s a risk that the “key insights” about trade-offs are partly a function of which specific models happened to be selected. Simpler approaches (e.g., learning a latent 3D representation from 2D video) might bridge the gap but aren’t explored here.

Experimental integrity: The scale is impressive (198 objects, 1980 sequences), but I’d want to see more on inter-annotator consistency and tracking failure modes. How often does the markerless pipeline fail? What’s the noise level in the extracted 3D geometry? The robot planning demo is described as “preliminary” — fair enough, but it’s thin evidence for the “real-world applicability” claim.

Writing quality: The abstract is clear and well-structured. However, the paper’s treatment of the tracking pipeline could use a dedicated comparison with marker-based baselines to establish credibility. The related work section likely glosses over some relevant visuotactile datasets (GelSight family, for instance). A rewrite of the limitations section — being more explicit about failure modes — would elevate trust.

Verdict: weak accept — The dataset fills a real need and the systematic comparison is valuable, but the technical novelty within the pipeline is incremental and the analysis, while insightful, feels like a first pass rather than a definitive study.

Takeaways

1. If you’re building deformable world models, you now have a real benchmark. Stop training on synthetic data or small lab setups. Deform360 gives you 198 real objects with ground-truth 3D geometry — use it.

2. The markerless tracking pipeline is transferable. If you need dense 3D tracking for any multi-view setup without fiducial markers, this pipeline’s approach (multi-view reconstruction + tactile fusion) is a starting point worth borrowing.

3. The 2D vs 3D trade-off framing is useful for your own research. Next time someone asks “why not just use a video model?” or “why not just use particles?”, you can point them to Deform360’s data: neither wins outright. The answer is probably hybrid — and now we have the data to explore that.

4. Multi-view + tactile is the gold standard for deformable manipulation data. Cameras alone miss contact; tactile alone misses global shape. If you’re designing a data collection system, follow this hardware template.