DOMINO: Teaching Robots to Catch Moving Objects

Paper: 2603.15620 Authors: Heng Fang, Shangru Li, Shuhan Wang, Xuanyang Xi, Dingkang Liang, Xiang Bai Categories: cs.CV, cs.RO

The Gap

Vision-Language-Action models like RT-1, RT-2, and OpenVLA have reached impressive performance on static manipulation tasks—picking up stationary objects, placing them in fixed locations. But they collapse when targets move. A robot that can reliably grasp a cup on a table fails completely when that cup is sliding across the surface or being carried by a person.

The failure has two roots: (1) training data is overwhelmingly static, and (2) mainstream VLAs process single frames, making them blind to motion. They see snapshots, not trajectories. This isn’t a minor limitation—dynamic manipulation is fundamental to real-world deployment, from catching thrown objects to collaborating with moving humans.

Problem: VLAs fail on moving targets
   |
   v
Root causes: [No dynamic data] + [Single-frame perception]
   |
   v
Method: DOMINO dataset + PUMA architecture
   |                      (optical flow + prediction)
   v
Evidence: 6.3% improvement + transfer to static tasks
   |
   v
Conclusion: Dynamic training builds robust spatiotemporal representations

The Increment

One sentence: Before this paper, robots trained on static datasets couldn’t track or intercept moving objects; after, they can leverage motion history and prediction to manipulate dynamic targets, with learned representations that transfer back to static tasks.

Core Mechanism

PUMA (Prediction-Unified Manipulation Architecture) has two coupled subsystems. First, a history-aware perception module that computes optical flow—pixel-level motion vectors—between consecutive frames. This flow field gets encoded and fed into the model alongside RGB observations, giving it explicit access to “what moved where” rather than forcing it to infer motion from static frames.

Second, a prediction module built around “world queries”—learnable tokens that aggregate scene information and implicitly forecast future object states. These queries attend to both current observations and historical flow, then produce latent representations that encode short-horizon predictions. The action decoder consumes both current perception and these predictive queries to output robot commands.

The coupling is key: perception informs prediction (flow tells you what to forecast), and prediction guides action (you move toward where the object will be, not where it is now).

Frame sequence --> [Optical Flow] --> Motion encoding
       |                                    |
       v                                    v
   RGB frames ---------> [Encoder] --> Visual features
                              |              |
                              v              v
                         [World Queries] <---+
                         (prediction)
                              |
                              v
                         [Action Decoder] --> Robot commands

Think of PUMA like a driver merging onto a highway. You don’t just look at where other cars are right now (single-frame perception). You glance in your mirror to see how fast they’re approaching (optical flow = motion history). Then you mentally project where they’ll be in two seconds (world queries = prediction). Finally, you time your merge based on that forecast (action conditioned on prediction). Without the motion history, you’d misjudge speeds. Without the prediction, you’d aim for where cars were, not where they’ll be. PUMA couples these the same way: flow feeds prediction, prediction guides action.

Key Concepts

• Optical Flow: Imagine taking two photos of a busy street one second apart. Optical flow is a map showing how each pixel moved between those photos—a car might shift 50 pixels right, a pedestrian 10 pixels down. It’s the visual system’s way of making motion explicit rather than implicit. For a robot, this means the difference between seeing “a cup” and seeing “a cup moving left at 20 cm/s.” The flow field is computed automatically from frame pairs using classical computer vision algorithms, then encoded as an additional input channel.

• World Queries: These are learnable tokens (think of them as blank index cards) that the model fills in during training. Each query learns to aggregate specific information from the scene—one might track “where the target object will be,” another “how fast it’s moving.” They work through attention: queries look at visual features and flow, pull out relevant patterns, and produce compact summaries. The magic is they’re trained end-to-end with the action decoder, so they learn to encode exactly the predictive information needed for control, without explicit supervision on what to predict.

• Spatiotemporal Reasoning: Static reasoning is like looking at a photograph and answering “where is the cup?” Spatiotemporal reasoning is watching a video and answering “where will the cup be in one second?” It requires understanding both spatial relationships (the cup is near the table edge) and temporal dynamics (it’s sliding at a certain velocity). Most VLAs lack this because they process frames independently. PUMA builds it by combining historical motion (flow) with learned prediction (queries), creating representations that encode both space and time.

Framework Shift

Before (mainstream VLAs):          After (PUMA):

Single frame                       Frame sequence
     |                                  |
     v                                  v
[Vision Encoder]                  [Optical Flow] + [Vision Encoder]
     |                                  |
     v                                  v
[Language Fusion]                 [World Queries] (prediction)
     |                                  |
     v                                  v
[Action Decoder]                  [Action Decoder]
     |                                  |
     v                                  v
  Action                             Action

Static snapshot                   Motion-aware + predictive

From reactive perception to anticipatory control, the core shift is making motion and prediction first-class citizens in the architecture rather than hoping the model infers them from static frames.

Expert Assessment

Problem choice: This is a genuine gap, not manufactured. Dynamic manipulation is conspicuously absent from major VLA benchmarks, and the failure modes are real—robots that work in labs fail in homes because objects move. The timing is right: static manipulation is maturing, so pushing toward dynamics is a natural next frontier.

Method maturity: The approach is solid but not groundbreaking. Optical flow for motion and learned queries for prediction are both established techniques. The contribution is more in the integration and the dataset than in novel algorithmic insights. I’d want to see ablations on whether simpler alternatives (e.g., just stacking frames, or explicit trajectory prediction) were thoroughly explored. The 6.3% improvement is meaningful but modest—suggests the method helps but isn’t a paradigm shift.

Experimental integrity: The dataset contribution (DOMINO with 110K trajectories) is valuable and likely the paper’s strongest asset. The benchmark with hierarchical task complexity is well-designed. However, I’d scrutinize whether baselines were given fair access to temporal information—if they’re single-frame models by design, the comparison isn’t entirely apples-to-apples. The claim that dynamic training helps static tasks is interesting and, if robust, suggests the learned representations are genuinely more general.

Writing quality: The abstract is clear and sets up the problem well. I suspect the related work section could be tighter—papers like this often over-cite to establish credibility. The method section likely needs more justification for design choices (why world queries over explicit trajectory prediction? why optical flow over learned motion features?). If I were reviewing, I’d push for clearer ablations and failure case analysis.

Verdict: Weak accept — the dataset and benchmark are solid contributions that will enable future work, and the method shows consistent if incremental improvements, but the algorithmic novelty is limited and the gains are modest.

Takeaways

The dataset is the real steal here—if you’re working on dynamic manipulation, DOMINO gives you 110K trajectories to train on, which is orders of magnitude more than what existed before. The hierarchical task structure (simple to complex) is smart for curriculum learning.

Optical flow as an explicit input is a simple trick worth trying in other video-based control problems. It’s computationally cheap and makes motion information immediately available rather than forcing the model to learn it from scratch.

The finding that dynamic training transfers to static tasks is counterintuitive and practically useful. If you’re building a manipulation system, training on harder (dynamic) data might give you better generalization than training only on the target (static) distribution. This suggests dynamic data forces the model to learn more robust spatiotemporal features.

World queries as a prediction mechanism could adapt to other sequential decision problems where you need to forecast future states without explicit supervision—think autonomous driving (predicting pedestrian trajectories) or game playing (anticipating opponent moves).