Manas: Dexterous Manipulation of Articulated Tools

Paper: 2606.13677 Authors: Zhao-Heng Yin, Guanya Shi, Pieter Abbeel, C. Karen Liu Categories: cs.RO, cs.AI, cs.CV, cs.LG

The Gap

Existing dexterous manipulation research focuses almost exclusively on rigid objects. Work like Dactyl, DexPilot, or general RLO environments assume fixed-geometry tools. For articulated tools (scissors, pliers, shears, folding knives), two challenges compound: (i) the tool has internal degrees of freedom that change the effective geometry mid-manipulation, and (ii) contact-rich interactions required to both grasp and actuate the tool are notoriously hard to model or learn from scratch. Prior attempts either simplify the tool as a rigid body (losing articulation), require extensive manual keyframing per tool, or depend on high-fidelity simulation with domain randomization that still struggles with joint dynamics. The field needs a method that can handle arbitrary articulated tools with minimal human effort and transfer to real hardware without per-tool fine-tuning.

Problem: Articulated tool manipulation is hard  (contact + internal DoF)
           |
           v
Existing Assumption: You need task-specific RL or careful sim-to-real tuning
           |
           v
Mana's Insight: Frame it as animation (coarse-to-fine, procedural keyframes)
           |
           v
Method: Procedural grasp generation -> motion planning -> RL refinement
           |
           v
Evidence: Zero-shot sim-to-real on 4 tools (scissors, pliers, shears, folding knife)
           |
           v
Conclusion: A general, scalable pipeline exists without per-tool manual engineering

The Increment

One sentence: Before Manas, articulated tool manipulation required task-specific RL policies and per-tool sim-to-real tuning; after Manas, a single pipeline with <1 minute of specification per tool yields zero-shot real-world transfer.

Core Mechanism

Mana consists of three stages:

1. Procedural Grasp Keyframe Generation: Given the tool mesh and a user-specified functional affordance (e.g., “scissors cut”, identified by clicking two points), the system automatically generates a set of plausible grasps using heuristic contact-point sampling and kinematic feasibility checks. This outputs “grasp keyframes” — static hand poses that satisfy grasp stability and tool closure constraints.

2. Motion Planning with Animation Priors: Each keyframe defines a goal configuration. A trajectory is computed using a sequence of motion planning steps (RRT-Connect style) that interpolates between open-grasp, close-grasp, and tool-actuation poses. The “animation” metaphor comes in: just as an animator defines key poses and the software fills in the in-between frames, Manas treats the manipulation problem as interpolating between functional grasp states.

3. RL Fine-tuning in Simulation: The planned trajectory becomes a coarse reference. A small RL policy is trained (using PPO with domain randomization) to refine the trajectory, handle contact dynamics, and add robustness. The reward is simply tracking the reference motion while maintaining grasp stability — no task-specific reward shaping.

Data flows: mesh -> keyframe generator -> trajectory planner -> RL policy (trains with simulator) -> deployable policy on real robot.

[Stage 1: Keyframe Generator]
Tool Mesh + Affordance (2 clicks)
         |
         v
   Sample candidate grasps (heuristic)
         |
   Filter by kinematics + stability
         |
   Output: N grasp keyframes (pose arrays)
         |
         v
[Stage 2: Motion Planner (Animation Core)]
   Keyframe sequence (open -> close -> actuate)
         |
   Interpolate via RRT-Connect (joint space)
         |
   Output: Coarse trajectory (waypoints)
         |
         v
[Stage 3: RL Refinement (PPO)]
   Coarse trajectory + simulator
         |
   Train to track reference + maintain contact
         |
   Domain randomization (physics, delays)
         |
   Output: Final policy (neural net weights)

Structural Metaphor: Think of this as stop-motion animation puppetry. The tool is a puppet with joints. The animator (user) specifies the key poses: hand open, hand closed, tool actuated — just like an animator maps out keyframes of a puppet’s mouth opening. The software (Mana) then automatically generates all the in-between frames (motion planning) and also ensures the puppet’s limbs don’t clip through each other (RL refinement). The “coarse-to-fine” is exactly the animation workflow: rough block the key poses, then spline interpolation, then hand-tweak each frame for realism. The advantage is that you don’t need to teach the puppet new choreography for every scene — you just give it the key moments and the pipeline fills the rest.

Key Concepts

• Functional Affordance: A very minimal specification of what the tool should do. The paper reduces this to “click two points on the tool mesh to define the functional axis” (e.g., for scissors, click the pivot and the tip). This bypasses the need for full task definitions — you don’t need to say “cut along a line at 10 cm thickness”. Instead, the algorithm infers grasps that can bring the two blades together. It’s a pragmatic approximation that works because the downstream RL can adapt.

• Grasp Keyframe Viability Filter: Not all candidate grasps are usable. The filter checks: (a) the hand can close the tool’s active joint, (b) the hand- tool contact pair doesn’t self-intersect, (c) the tool’s Center of Mass is approximately enclosed by the hand’s contact polygon. This is computed in milliseconds using geometric heuristics — no expensive simulation needed.

• Animation-inspired Trajectory Refinement: Traditional manipulation pipelines (e.g., Task and Motion Planning) separate grasp planning from execution. Manas treats the entire manipulation as a sequence of keyframes, then uses RL to “smooth” the motion the way an animator would tweak curves. This is novel because it decouples the high-level structure (keyframes) from low-level contact dynamics (RL), allowing each stage to be simple.

Framework Shift

Before (mainstream approach):
   Task specification (e.g., "cut paper") -> hand-label robot motions
   -> per-task RL with task reward -> sim-to-real transfer (domain randomization)
   -> policy only works for that specific tool-task pair.
   [Each new tool requires new reward engineering + new sim tuning]

After (this paper):
   Tool mesh + 2 clicks (functional affordance) -> procedural keyframes
   -> motion planning -> RL refinement (general reward: track keyframes)
   -> policy that zero-shot transfers to real robot for that tool.
   [Same pipeline works for any articulated tool with no extra engineering]

One sentence: From task-specific reward engineering and per-tool sim tuning to a unified animation-based pipeline where only a few mouse clicks define the affordance, the core shift is replacing task decomposition with keyframe interpolation.

Expert Assessment

Problem choice: Real gap. Articulated tools are everywhere (scissors, pliers, wrenches, folding knives, etc.) and prior work has systematically ignored them because they’re inconvenient to model. The problem sits exactly where robotics needs to go next — handling everyday articulated objects. Not manufactured.

Method maturity: Clever insight, not brute force. The animation analogy is the real contribution — it’s a framing that simplifies the problem without hand-engineering. The procedural keyframe generation is elegant (geometric heuristics, no sim required). The RL refinement is standard (PPO + domain randomization), but that’s appropriate: the novelty is in how the problem is decomposed, not in the RL itself. Simpler approaches like pure motion planning would fail due to contact dynamics; the coarse-to-fine is the right compromise.

Experimental integrity: Baselines are fair: they compare against a naive motion planning approach (no RL) and a direct sim-to-real method without the keyframe pipeline. The zero-shot transfer results on four tools are convincing — videos show real robot successfully cutting paper with scissors, gripping with pliers, etc. Numbers are reported as success rates over multiple trials. Red flag: only four tools, all with one degree of freedom. Would be more convincing with more complex articulated tools (e.g., multi-joint pliers, compound tools). Also, the “zero-shot” sim-to-real depends on domain randomization — but the paper doesn’t ablate the importance of domain randomization vs. the keyframe pipeline. Slight lack of ablation.

Writing quality: Clear, well-structured. The metaphor is explained well. The weakness is in the experimental section: they don’t report failure cases or edge cases. If they rewrote Section IV (Experiments) to include a “failure analysis” paragraph, it would elevate the paper. The abstract is slightly too vague on “few mouse clicks” — the paper should show a screenshot.

Verdict: strong accept — Clean framing, solid engineering, convincing results on an underexplored problem. The animation metaphor is likely to influence future work.

Takeaways

• Use functional affordances, not full task specs. By specifying only the tool’s actuation axis (two clicks), you avoid task-specific reward design. This trick can generalize to other manipulation problems (e.g., opening a bottle: click cap top and bottle neck).
• Coarse-to-fine with animation keyframes. Instead of solving full trajectory optimization end-to-end, break the problem into a rough plan (keyframes + interpolation) and refine with RL. This principle applies to any complex motor skill: start with a good guess, then learn local corrections.
• Geometric heuristic for grasp viability is surprisingly effective. You don’t need deep learning for grasp selection if you know the tool’s kinematic structure. This suggests a design principle: when possible, exploit geometry before resorting to data-driven methods.
• For practitioners: the code (assuming it’s released) likely contains a modular keyframe generator and motion planner that can be repurposed for other articulated objects. Steal the “click two points” interface — it’s the minimum viable specification for tool function.