AFUN: Teaching Robots Where and How to Touch

Paper: 2606.02551 Authors: Zhaoning Wang, Yi Zhong, Jiawei Fu, Henrik I. Christensen, Jun Gao Categories: cs.RO, cs.CV

The Gap

Affordance research has split into two camps: methods that localize task-relevant regions (where to interact) without specifying executable motion, and methods that predict motion but don’t scale across diverse objects and tasks. Prior work like AffNet and AffordanceLLM segments interaction areas but leaves the robot guessing how to move. Motion prediction methods like Contact-GraspNet output grasp poses but struggle to generalize beyond their training distribution. No existing model answers both “where” and “how” while scaling to open-world diversity.

Problem: Affordance = where + how, but existing work picks one
   |
   v
Assumption: A shared representation can encode both spatial regions
            and object-centric motion if trained on diverse data
   |
   v
Method: Predict functional mask (where) + 3D motion curve (how)
        from RGB-D + language, trained on unified multi-source data
   |
   v
Evidence: +23.9 gIoU on segmentation, +12.7-61.3% hit-rate on contact,
          best motion prediction, zero-shot robot deployment works
   |
   v
Conclusion: Joint prediction + diverse training → open-world affordance

The Increment

One sentence: Before AFUN, robots could see where to interact OR guess how to move; after AFUN, they predict both from a single observation, generalizing across objects and tasks without task-specific tuning.

Core Mechanism

AFUN takes an RGB-D image and a language task description as input. A vision-language encoder processes both modalities into a shared representation. This representation feeds two parallel prediction heads: one outputs a 2D task-conditional functional mask indicating where interaction should happen, the other predicts a 3D post-contact motion curve in object-centric coordinates describing how to move after contact.

The model is trained on a standardized data pipeline that converts heterogeneous sources—robot demonstrations, human videos, simulation trajectories, 3D scans—into a unified affordance schema. Each data point gets a language label, a functional mask, and a 3D motion trajectory normalized to object coordinates. The pipeline handles scale, viewpoint, and embodiment variation by projecting all motions into a canonical object-centric frame.

During inference, given a novel object and task, the mask head segments the interaction region using cross-attention between visual features and language embedding. The motion head samples trajectory waypoints from learned distributions conditioned on the mask region and task semantics. The predicted motion is object-centric, so it can be transformed to any robot’s coordinate frame without retraining.

RGB-D + Language
      |
      v
  [Encoder] --> Shared Representation
      |
      +----------+----------+
      |                     |
      v                     v
[Mask Head]           [Motion Head]
      |                     |
      v                     v
2D Functional Mask    3D Motion Curve
(where to touch)      (how to move)

Think of AFUN as a bilingual translator with specialist interpreters. The encoder is the translator who converts visual scenes and language into a common internal language—the shared representation. From this internal language, two specialist interpreters work in parallel. The mask interpreter reads the internal language and marks regions on a 2D map of the scene. The motion interpreter reads the same internal language and draws a 3D path in object space. Both interpreters reference the same source material but produce different outputs optimized for their domain. Because the internal language is object-centric (like describing “the handle” rather than “37cm from the robot”), the interpreters’ outputs work across different cameras, robots, and viewpoints—they’re describing the object’s inherent properties, not observer-dependent coordinates.

Key Concepts

• Object-centric motion representation: Instead of recording trajectories in robot or camera coordinates (which change with viewpoint and embodiment), AFUN represents motion relative to the object being manipulated. Imagine you’re teaching someone to open a drawer. You don’t say “move your hand 20cm forward relative to where you’re standing”—that only works if they stand exactly where you stood. Instead, you say “pull the handle straight out along the drawer’s axis.” That description works regardless of where they stand or how tall they are. AFUN does the same thing: it learns to predict motion in the object’s frame (“move along the handle axis”) rather than the robot’s frame, so the same learned motion transfers to new viewpoints and robot morphologies.

• Task-conditional functional mask: Not all parts of an object are relevant for every task. A mug has a handle, a rim, and a body—which region matters depends on whether you’re grasping, pouring, or stirring. The functional mask is a 2D segmentation that highlights task-relevant regions given a language description. It’s not a generic saliency map; it’s conditioned on what you want to do. If you say “grasp,” it segments the handle. If you say “pour,” it segments the rim. This conditioning happens through cross-attention between visual features and language embeddings, so the mask adapts to novel task descriptions without retraining.

• Affordance schema unification: Different data sources record affordances in incompatible formats: robot logs have joint angles, human videos have pixel motion, simulations have object poses. AFUN’s data pipeline converts all sources into a shared schema: language label, 2D mask in image space, 3D trajectory in object space. This unification enables training on millions of diverse examples—robot demos provide accurate contact points, human videos provide natural motion diversity, simulations provide dense coverage of object categories, 3D scans provide geometric grounding. The schema acts as a common currency, letting the model learn from heterogeneous data without source-specific preprocessing at inference time.

Framework Shift

Before (mainstream approach):        After (this paper):

Scene --> [Detector] --> Mask        Scene + Task --> [Encoder]
            (where)                                       |
                                                    +-----+-----+
Motion --> [Planner] --> Trajectory                |           |
            (how)                              [Mask Head] [Motion Head]
                                                    |           |
Disjoint pipelines,                            Where + How in one shot,
limited generalization                         open-world generalization

From sequential problem-solving (first segment, then plan motion) to joint prediction, the core shift is treating affordance as a unified prediction task where spatial and motion reasoning inform each other through shared representation.

Expert Assessment

Problem choice: Real gap, not manufactured. The field has been stuck at this bifurcation for years—you either get pretty segmentations or brittle motion predictions, rarely both. Positioning this as a “foundation model” step is ambitious but justified given the scale and diversity of the data pipeline. The problem sits at the inflection point where enough data and compute exist to attempt unification.

Method maturity: Solid engineering over conceptual novelty. The core insight—predict mask and motion jointly from shared representation using diverse data—is straightforward. The real contribution is the data pipeline that makes this tractable. The object-centric motion representation is borrowed from prior work but applied systematically here. No magical architecture; the wins come from scale and data curation. That said, the paper doesn’t overclaim—it’s honest about being “a step towards” rather than the final solution.

Experimental integrity: Baselines are fair and comprehensive. The improvement margins (+23.9 gIoU, +12.7-61.3% hit-rate) are large enough to survive scrutiny. I’d like to see failure case analysis—when does the object-centric assumption break down? The real-world robot experiments are convincing but limited to structured tasks. Would’ve been stronger with quantitative metrics on real robot success rates rather than qualitative demos.

Writing quality: The abstract and introduction are crisp. Section 3 (method) gets dense—the data pipeline description would benefit from a figure showing the conversion process for each source type. The discussion section is thin; missed opportunity to reflect on what the model hasn’t learned (e.g., multi-object interactions, tool use, deformable objects). Supplementary material likely carries important details that should be in the main text.

Verdict: weak accept — the data unification is valuable infrastructure for the field, results are strong across benchmarks, but the method itself is incremental and the writing could better contextualize limitations.

Takeaways

Steal the object-centric motion trick: If you’re building any system that needs to generalize across viewpoints or embodiments, represent actions relative to the manipulated object, not the agent. This applies beyond robotics—3D scene editing tools, AR interfaces, even game AI can benefit from canonical object frames.

Multi-source data unification pays off: The schema-based pipeline that converts heterogeneous data into a shared format is more reusable than the model architecture. If you’re in a domain with fragmented data sources (medical imaging, geospatial, multimodal retrieval), invest in a unifying representation before scaling models.

Joint prediction beats sequential pipelines for coupled problems: When two predictions inform each other (here, where and how), training them jointly through a shared representation outperforms training them separately and stitching outputs. The principle transfers: in document understanding, jointly predict layout + semantics; in dialogue, jointly predict intent + response structure.