What Matters, When? Distractors Break the Target Choice, Not the Skill

Paper: 2609.05376 Authors: Vivek Chavan, Pengtao Xie, Yahuan Shi, Oliver Heimann, Kevin Haninger, Jörg Krüger Categories: cs.AI, cs.CV, cs.RO

The Gap

Visuomotor imitation policies reach high performance under in-distribution visual conditions and then fail when visually similar objects or receptacles are introduced. That failure is easy to observe and easy to misattribute — it looks like a capability problem, as if the policy had not learned to manipulate well enough.

The paper’s diagnosis is different and more specific: it is a problem of conditional visual grounding. The visual target required for successful control changes with the manipulation phase, and in more complex tasks with the observed task state. So the policy must do two things: identify the right target, and do so conditionally on where it is in the task. A distractor breaks the first without necessarily damaging the second — which is why the failure looks like incompetence while the skill is intact.

That distinction has a practical consequence. If the failure is in target selection, the remedies are about attention and grounding. If it were a skill failure, the remedy would be more demonstrations of the manipulation itself — a much more expensive fix aimed at the wrong thing.

   IN-DISTRIBUTION SUCCESS, OUT-OF-DISTRIBUTION FAILURE

   visuomotor imitation policies perform WELL in-distribution
   and FAIL when VISUALLY SIMILAR objects or receptacles appear
        |
        v
   [EASY TO MISATTRIBUTE]
     it LOOKS LIKE a capability problem, as if the policy had
     not learned to manipulate well enough
        |
        v
   [THE PAPER'S DIAGNOSIS, MORE SPECIFIC]
     a problem of CONDITIONAL VISUAL GROUNDING
       the visual target required for successful control
       CHANGES WITH THE MANIPULATION PHASE
       and in more complex tasks WITH THE OBSERVED TASK STATE
        |
        v
   so the policy must do TWO things:
     [1] identify the RIGHT TARGET
     [2] do so CONDITIONALLY on where it is in the task
        |
        v
   a distractor breaks [1] WITHOUT necessarily damaging [2]
     -> the failure looks like INCOMPETENCE while the SKILL
        IS INTACT
        |
        v
   [PRACTICAL CONSEQUENCE]
     if the failure is in TARGET SELECTION: remedies are about
       ATTENTION and GROUNDING
     if it were a SKILL failure: the remedy would be MORE
       DEMONSTRATIONS of the manipulation -- a much more
       expensive fix aimed at the WRONG THING

The Increment

One sentence: Before this paper, distractor-induced failure looked like a manipulation weakness; after it, localising it to picking and placement shows the sensitivity depends on the kind of similarity and the task stage, and three complementary interventions aimed at target selection recover robustness on real hardware.

Core Mechanism

The design earns the diagnosis by making the failure measurable rather than anecdotal.

Systematic distractors with controlled color and shape similarity, using Action Chunking with Transformers (ACT) as the policy. Controlling the similarity along named axes is what makes the finding interpretable: “distractors hurt” would be uninformative, whereas similarity type is a variable you can act on.

Failures are localised to picking and placement. This is the step that establishes the diagnosis. If the policy were failing to manipulate, errors would spread across the whole trajectory; concentrating them at the two stages where target choice is the deciding act is what supports “conditional visual grounding” over “weak manipulation skill”.

And the sensitivity is structured: distractor sensitivity is specific to both the type of visual similarity and the manipulation stage. Both dependencies matter. Type-specificity means shape and colour distract differently, so a single augmentation strategy will not cover both. Stage-specificity is the more interesting one — it means the same distractor can be harmless at one point in the task and fatal at another, which follows directly from the grounding being conditional. A policy that grounds its target on the right cue at the wrong time fails.

Guided by that diagnosis, three complementary interventions are evaluated, each aimed at target selection while preserving spatial information required for control:

• Distractor augmentation — the data-side fix.
• Phase-dependent attention regularization — the architectural fix, and its phase-dependence is what the diagnosis bought: attending correctly is not a single property but one that varies across the task.
• Appearance-based visual prompting — an inference-time fix, supplying the appearance information the policy needs to disambiguate.

The qualifier “while preserving spatial information required for control” is important and easy to skip. Improving target selection by degrading spatial precision would trade one failure for another, so the interventions are measured on robustness rather than on distractor immunity alone.

These interventions substantially improve robustness in simulation and on a physical UR3e. Real hardware is the check that matters for a manipulation claim, and the paper supplies it.

The final element broadens the claim beyond imitation policies. The same failure pattern is examined in a pretrained vision-language-action policy on a state-conditioned instrument-handling task, where the observed state of a medical instrument determines the correct destination. That is a different policy class — pretrained VLA rather than ACT — and a different conditioning structure, since the target depends on the instrument’s state rather than on the phase number. Finding the same pattern there supports the general claim: visual distractors can cause incorrect object or destination selection even when the underlying manipulation skill remains intact, and explicitly improving target selection substantially recovers performance across distinct visuomotor policy-learning regimes.

   DESIGN: MAKE THE FAILURE MEASURABLE

   ACT (Action Chunking with Transformers) as the policy
   SYSTEMATIC distractors with CONTROLLED colour and shape
     similarity
       <- controlling similarity along NAMED AXES makes the
          finding interpretable: "distractors hurt" is
          uninformative; similarity TYPE is a variable you
          can ACT ON

   LOCALISATION: failures concentrate at PICKING and PLACEMENT
     <- establishes the diagnosis: were the policy failing to
        MANIPULATE, errors would spread across the whole
        trajectory; concentrating them at the two stages where
        TARGET CHOICE is the deciding act supports
        "conditional visual grounding" over "weak skill"

   THE SENSITIVITY IS STRUCTURED
     specific to the TYPE of visual similarity
       -> shape and colour distract differently, so a single
          augmentation strategy will not cover both
     specific to the MANIPULATION STAGE
       <- the more interesting one: the same distractor can be
          harmless at one point and FATAL at another, which
          follows directly from the grounding being CONDITIONAL
       -> a policy that grounds on the right cue at the WRONG
          TIME fails

   THREE COMPLEMENTARY INTERVENTIONS, each aimed at target
   selection WHILE PRESERVING the spatial information
   required for control
     distractor AUGMENTATION          (data side)
     PHASE-DEPENDENT attention regularization (architecture;
       its PHASE-DEPENDENCE is what the diagnosis bought)
     appearance-based VISUAL PROMPTING (inference time)
     <- the qualifier matters: improving target selection by
        DEGRADING spatial precision would trade one failure
        for another

   RESULT: substantial robustness gains in SIMULATION and on a
           PHYSICAL UR3e

   BROADENING: the same pattern appears in a PRETRAINED
   vision-language-action policy on a STATE-CONDITIONED
   instrument-handling task (the observed state of a medical
   instrument determines the destination)
     <- a different policy class, and a different conditioning
        structure (STATE rather than PHASE)
     -> supports the general claim: distractors can cause wrong
        object or destination selection even when the SKILL is
        intact, and improving target selection recovers
        performance ACROSS distinct regimes

Think of it as a surgeon who can operate perfectly and cannot tell which of two identical instruments is on the tray. The technical skill is untouched — the hands know what to do — but the operation fails because the wrong instrument gets picked up, and the failure looks like incompetence in the notes. Two details from the paper make the metaphor exact. The instruments look similar in specific ways (colour, shape), and which similarity causes the confusion depends on both the cue and the moment: at the start, choosing between two similar trays matters; mid-procedure, when the destination depends on the instrument’s state, the same similarity matters differently. And the fixes are all about making the right object identifiable — labelling it, training attention to where it matters at each stage — rather than re-teaching the operation.

Key Concepts

• Conditional visual grounding: the target changes with the phase, and with observed task state in harder tasks. It converts “the policy failed” into a specific requirement — identify the right target, conditionally on task progress.
• Localisation as the diagnostic: concentration of failures at picking and placement, the two stages where target choice decides. It is what distinguishes a grounding failure from a skill failure.
• Structured sensitivity: dependence on both similarity type and manipulation stage. The stage-dependence follows from conditionality and implies that a single augmentation cannot cover the space.
• Interventions that preserve spatial information: augmentation, phase-dependent attention regularization, and appearance prompting, all measured on robustness. Target selection improved by blurring spatial precision would not be an improvement.
• Cross-regime replication: the same pattern in a pretrained VLA on a state-conditioned task. It supports the general claim across policy classes and conditioning structures.

Framework Shift

Before (distractor failure as a capability gap):
  policy works in-distribution, fails with similar objects
  -> attributed to weak manipulation learning
  -> remedy: more demonstrations of the manipulation
  -> expensive fix aimed at the wrong component

After (distractor failure as conditional grounding):
  distractors with controlled similarity; failures localised
    to picking and placement
  -> sensitivity depends on similarity TYPE and task STAGE
  -> interventions target SELECTION while preserving spatial
     information
  -> robustness recovered in simulation and on a UR3e, and the
     pattern replicates on a pretrained VLA

From treating a robustness failure as a general weakness, to localising it to the target-selection step and conditioning the remedy on task phase, the core shift is that the skill and the choice of what to apply it to are separate, and only the second is broken.

Expert Assessment

Problem choice: Excellent, and the diagnostic framing is what makes it useful. Distractor failures are a well-known annoyance in robotics, and the default response — collect more demonstrations, augment harder, add data — is expensive and often aimed at the wrong component. Localising the failure to a decision rather than a skill changes what to do about it.

Method maturity: The design is clean: controlled similarity along named axes, localisation to specific stages, and interventions that are each attributable to a different layer of the stack (data, architecture, inference). The insistence on preserving spatial information in the interventions is the detail that separates a real improvement from a trade. Replicating the pattern on a pretrained VLA with a state-conditioned rather than phase-conditioned target is the strongest generalisation evidence, because it tests a different conditioning structure rather than the same one again.

Experimental integrity: Validating on a physical UR3e is the check that matters for a manipulation claim, and the phase- and type-specificity findings are reported as structure rather than as a single effect size, so they can be acted on. The limitation is that ACT is a comparatively simple imitation policy, and while the VLA replication broadens the claim, both settings share the paper’s controlled distractor construction — how these failure rates scale with realistic visual clutter, rather than constructed distractors, is left to future work.

Writing quality: The title poses the right question — what matters, when — and the body answers it with the stage-specificity finding, which is the most useful part. Because the remedies are concrete and layered, a short summary translating the diagnosis into a checklist — check whether errors concentrate at picking and placement, then apply the corresponding intervention — would make it directly usable by a team debugging a deployed policy.

Verdict: strong accept — it reclassifies a common robustness failure from a capability problem to a target-selection problem, localises it by stage, and shows layered interventions recovering performance on real hardware.

Takeaways

• Localise before concluding incompetence. If failures concentrate where a choice is made rather than along the whole skill, the problem is grounding, not capability.
• Check whether sensitivity is stage-dependent. A conditional target means the same distractor can be harmless early and fatal later.
• Improve target selection without degrading spatial precision. Otherwise the fix trades one failure for another, and robustness measures catch that where distractor immunity would not.
• Test on a different conditioning structure, not just a different architecture. Replicating on a state-conditioned task is what establishes generality here.