DexCompose: Reusing Dexterous Policies for Multi-Task Manipulation with a Single Hand

Paper: 2606.28323 Authors: Dihong Huang, Zhenyu Wei, Zhuxiu Xu, Yunchao Yao, Sikai Li, Mingyu Ding Categories: cs.RO, cs.AI, cs.CV, cs.LG

The Gap

Dexterous hands can learn impressive single skills (e.g., grasping a bottle, pushing a button, rotating a cap). But when you ask the same hand to simultaneously hold a bottle and push a button, the two skills fight over the same fingers – a conflict of contact modes, forces, and finger poses. Prior work either chains policies sequentially (which breaks the first skill while executing the second) or trains monolithic multi-task policies from scratch (expensive, brittle, and doesn’t reuse existing skills). The core boundary: no one had shown how to compose two full-hand dexterous policies without destructive interference at the finger level.

[Problem] ----------> [Assumption + Method]
    |                        |
    | (Interference)         | (DexCompose: finger-level ownership)
    v                        v
[Evidence 1] ------------> [Evidence 2]
(naive chaining fails)    (77.4% composite success)
    |
    v
[Conclusion] Interference can be avoided by identifying essential fingers 
            and restricting the new policy to non-essential subspace.

The Increment

One sentence: Before this paper, composing two dexterous policies with a single hand was unreliable (sub-20% success). After this paper, the same hand can reliably combine skills (77.4%) by explicitly assigning action ownership to individual fingers.

Core Mechanism

DexCompose works in three stages. Stage 1 – Release Tests: Given a pretrained skill A (e.g., hold a bottle), execute it to a stable state. Then, for each candidate finger mask (subset of fingers), zero out the actions of those fingers and check if the skill state collapses. This identifies the essential finger set – fingers whose movement would destroy the first skill.

Stage 2 – Dual Residuals: Two small residual modules are trained while freezing the original policies.

• A bounded residual stabilizer takes the current state and outputs small corrective actions for the essential fingers. It ensures that the first skill’s state does not drift when the other fingers move.
• A context-aware residual wraps the second policy. It observes the current state (including the first skill’s object pose) and adjusts the second policy’s output only for the non-essential fingers. The second policy’s original actions for essential fingers are zeroed out.

Stage 3 – Deployment: The hand runs the first policy + stabilizer on essential fingers, and the adapted second policy on non-essential fingers. The two streams run concurrently without conflict.

[Pretrained Policy A]      [Pretrained Policy B]
         |                         |
         v                         v
   [Release Tests]                 |
         |                         |
   [Essential Finger Mask]         |
         +-------------------------+
         |                         |
         v                         v
 [Bounded Stabilizer]    [Context-Aware Residual]
  (essential fingers)    (non-essential fingers)
         |                         |
         +----> [Composite Action] <----+
                     |
                     v
             [Robot Hand]

Structural Metaphor: Think of a pianist playing a sustained chord with one hand while a second melody must be played by the same hand. The pianist first figures out which fingers are absolutely required to hold the chord (release tests: try lifting each finger and see if the chord sounds wrong). Then, while keeping those fingers rigidly in place (stabilizer), they use the remaining free fingers to tap the new melody (residual adaptation). The stabilizer is a tiny, automatic correction that prevents the held fingers from wobbling, while the melody fingers are free to move independently. The “conductor” (DexCompose) never asks a finger to do two things at once—each finger has a single, clear role.

Key Concepts

• Release Tests: A brute-force but effective way to discover which action dimensions (finger joints) are critical for a skill. You simply run the policy, then set certain action outputs to zero and observe whether the skill state collapses. The set of fingers that, when zeroed, cause failure are the “essential” ones. This avoids any prior assumptions about finger roles.

• Action Subspace: The space of all possible hand actions is high-dimensional. By restricting the second policy to only act in the subspace of non-essential fingers, you guarantee zero interference with the first skill. The stabilizer only operates in the essential subspace. This decomposition is the key structural innovation.

• Bounded Residual: A residual correction that is guaranteed to be small (bounded in magnitude). It nudges the essential fingers just enough to maintain the first skill’s state, without overriding the original policy’s intention. Without the bound, the stabilizer might learn to “distort” the original skill.

Framework Shift

Before (naive chaining or joint training):         
 
 [Policy A] --> [State] --> [Policy B] --> [Mixed] 
              (loss of first skill)

After (DexCompose):                                
 
  [Policy A] -+--> [Stabilizer] -+--> [Essential Fingers]
              |                   |
              +--> [Frozen]       +--> [Non-Essential Fingers]
  [Policy B] -+--> [Residual] ----+

From naive sequential rollout or monolithic multi-task training to finger-level action decomposition with dual residuals, the core shift is ownership: each finger is exclusively assigned to either maintaining the first skill or executing the new task, preventing any conflict.

Expert Assessment

Problem choice: Real gap. Composing dexterous skills is the next bottleneck in dexterous manipulation; the community has focused on single skills or simple stacking. This paper targets a precise and well-motivated failure mode.

Method maturity: Clever but not trivial. The release test is a simple insight (systematic ablation of finger actions), and the dual residual architecture is a natural follow-up. There might be simpler heuristics (e.g., using a mask from force sensors), but the paper’s approach is principled and domain-agnostic.

Experimental integrity: Fair. Baselines include finetuning, policy chaining, and a naive residual without finger decomposition. The composite tasks (4 retention skills × 4 interactions) cover a good range. Success rates are clear and consistent. No obvious red flags, though the release test requires success on the first skill – if skill A is unrepeatable, the method breaks (they note this).

Writing quality: Solid but verbose. The method description could be tighter – the release test details are explained twice. The “related work” section is a bit thin. If the authors rewrote Section IV with a single clean algorithmic block, the paper would be easier to follow.

Verdict: weak accept – a clean, practical solution to a narrow but important problem. Not a paradigm shift, but a well-executed incremental step that the field can build on.

Takeaways

Practitioners can steal two specific ideas:

1. Use systematic finger zero-out tests to discover skill-critical action dimensions – this transfers to any multi-dimensional control problem (not just dexterous hands). You can identify which control signals are essential and which are free.
2. Apply bounded residuals on essential dimensions combined with unrestricted residuals on non-essential dimensions – a general template for composing any two full-body policies without retraining. For example, combine a locomotion policy with an arm manipulation policy on a humanoid robot by identifying which leg joints are critical for balance.

If the release tests are cheap (as they are in simulation), this is a low-cost drop-in solution for multitask robotics.