Diagnosing Robot Language Understanding: Fixing Compositional Generalization with Bias-Aware Data Collection

Paper: 2607.21582 Authors: Yu Qi, Zhang Ye, Xinyi Xu, Yuxuan Lu, Amitoj Sandhu, Boce Hu, Haojie Huang, Jonathan Tremblay, Lawson L. S. Wong Categories: cs.RO, cs.CV

The Gap

Existing research knows that fine-tuned robotic policies are “lazy”—they take shortcuts by over-relying on a few salient cues (like object color) rather than truly understanding the full composition of a language instruction. However, this understanding was largely anecdotal or came from error analysis after a policy fails. There was no precise, quantitative diagnostic tool to localize the failure to specific semantic components of the instruction before deployment. This paper builds that tool. The logical path is clear:

Problem: Policies take shortcuts on compositional instructions.
  |
  v
Assumption: These shortcuts can be decomposed into bias toward individual
            instruction factors (color, verb, object, size, spatial).
  |
  v
Method: Develop metrics (FDR, FDH) to quantify these factor biases.
  |
  v
Evidence: Apply diagnostics to 6 foundation models, revealing a consistent
          bias hierarchy (color > object > spatial > verb > size).
  |
  v
Conclusion: Diagnosis is actionable. Collecting more data targeting the
            weakest (most under-grounded) factors yields better policies.

The Increment

One sentence: Before this paper, we knew robots were biased in how they understood language; after this paper, we have a precise metric to measure that bias per semantic component and a principled way to fix it through targeted data collection.

Core Mechanism

The method has three core stages. First, it decomposes any given instruction (e.g., “pick up the small red block”) into its constituent factors: verb: pick, object: block, color: red, size: small. This creates a structured, factor-level representation. Second, it quantifies bias by systematically testing the policy. For a pair of factors, like color and object, it finds cases where the policy must choose between following one or the other. The Factor Dominance Rate (FDR) measures how often the policy defers to one factor over the other. Aggregating these pairwise comparisons creates the Factor Dominance Hierarchy (FDH), a global ranking of which factors the policy trusts most. Third, it uses this diagnosis to strategically collect data. Instead of collecting demonstrations uniformly, it allocates a fixed budget of effort to create more examples that specifically challenge the policy’s weakest (most under-grounded) factors.

[Instruction] --> [Factor Decomposition] --> \{verb, object, color, size, spatial\}
                                                 |
                                                 v
[Policy] <----- [Bias Quantification] <----- {Factor Pairs}
    |                  |                          |
    v                  v                          v
[Action]      [FDR: pairwise bias]     [FDH: global hierarchy]
                                                 |
                                                 v
                                          [Diagnosis: color strong, verb weak]
                                                 |
                                                 v
                                      [Data Collection Strategy]
                                      Allocate budget to make more
                                      verb/spatial/size-focused demos.

Think of it like a team of specialist chefs being evaluated on complex dishes. An instruction is a recipe with components: a technique (verb), an ingredient (object), a sauce (color), a portion size (size), and a plating instruction (spatial). You suspect the team just memorizes the sauce. To diagnose, you give them conflicts: a recipe that says “use the red sauce” but shows a picture with green sauce. You check if they follow the recipe text (color) or the picture (object). You repeat this for all component pairs. You discover they blindly follow the sauce (color) 90% of the time, but only follow the technique (verb) correctly 40% of the time. That’s your FDR and FDH. The actionable fix isn’t just to cook more dishes (collect more data); it’s to specifically design and practice dishes where the technique and portion size are the critical, tricky parts. You train the chefs where they are weakest, not just where they are already strong.

Key Concepts

• Instruction Factor: This is the paper’s atomic unit of meaning in a command. Imagine you’re giving a robot a task: “Put the blue cup on the large plate.” The factors are the reusable, semantic slots: *verb (put), object 1 (cup), attribute 1 (blue), object 2 (plate), attribute 2 (large). The core idea is that a robot can get the right answer by only listening to one slot (e.g., always grabbing whatever is blue) without truly understanding the whole structure. Breaking it down lets us measure which slots it’s actually listening to.
• Factor Dominance Rate (FDR): This is a pairwise bias score. Let’s say we test Color vs. Verb. We create a scenario where the instruction says “push the red block” (Verb: push, Color: red), but only a blue block is present. If the robot tries to push the red block (which doesn’t exist) or does nothing, it’s following Color. If it pushes the blue block, it’s following Verb. The FDR for Color > Verb is the percentage of times the robot follows the color cue over the verb cue in such conflicts. It’s a direct measurement of one factor’s “authority” over another in the robot’s learned policy.
• Factor Dominance Hierarchy (FDH): This is the aggregated league table from all the pairwise FDR scores. If Color beats Object, Spatial, Verb, and Size in pairwise contests, and Object beats Spatial, Verb, and Size, and so on, you get a rank order. The paper’s key finding is that across six different foundation models, this ranking is surprisingly consistent: Color is the dominant shortcut, Verb and Size are the most poorly grounded, and Object and Spatial fall in between. This isn’t just a list of failures; it’s a structural diagnosis of the model’s understanding hierarchy.

Framework Shift

Before (mainstream approach):        After (this paper):
[Collect lots of diverse demos]      [Diagnose factor bias first]
          |                                    |
          v                                    v
[Hoping coverage fixes all bugs]     [Identify weak factors (e.g., verb, size)]
          |                                    |
          v                                    v
[Uniform improvement, blind to       [Strategically collect demos targeting
 underlying bias structure]           weak factors to break shortcuts]

From hoping for coverage to engineering for weakness, the core shift is treating data collection as a targeted, diagnostic-driven intervention rather than a generic scaling effort.

Expert Assessment

Problem choice: Excellent. The “shortcut learning” problem in visuomotor policies is real and well-known. This paper refines it from a vague observation into a precise, measurable phenomenon at the factor level. It sits squarely at the intersection of robotic learning, language grounding, and fairness/bias in AI, making it highly relevant.

Method maturity: Clever and practical, not brute force. The idea of creating structured conflicts to measure bias is powerful. Using LLMs for factor decomposition is a smart, scalable lever. The data collection strategy is elegantly simple: just focus on the diagnosed weak points. A simpler approach might be to just uniformly increase data on the worst-performing task categories, but this paper’s method is more fundamental by targeting the *semantic components of understanding, not just task performance.

Experimental integrity: The baselines seem fair—comparing against standard uniform data collection. The real-world experiment using *half the demonstrations to achieve better performance is a strong, convincing result. A minor flag: the diagnosis assumes factor independence and additive biases, which might not hold perfectly in complex, interacting instructions. But as a first-order approximation, it’s robust.

Writing quality: Generally clear and well-structured. The weakest section is the Related Work, which could more sharply contrast the specific contribution (quantitative, per-factor diagnosis) with prior art on compositional generalization (which often focuses on new architectures or training objectives). The conclusion could also dig deeper into the limits of the factor decomposition assumption.

Verdict: weak accept — It introduces a genuinely useful diagnostic lens and a simple, effective data collection principle, backed by solid simulation and real-robot experiments.

Takeaways

1. Steal the diagnostic framework: Even if you don’t use their exact factors, the idea of breaking down your model’s input (text, image features) into semantic components and creating controlled conflict scenarios to measure pairwise bias is a powerful evaluation technique. It can be applied to debug multimodal models in other domains (e.g., “Does the model prioritize the caption or the visual style?”).
2. Steal the “diagnose-then-collect” paradigm: Stop just collecting more random data. Use a small, diagnostic dataset to find your model’s specific semantic blind spots (e.g., it confuses “left” and “right” or ignores numbers). Then, design your larger data collection or generation effort to explicitly and repeatedly target those blind spots. This is a far more efficient use of annotation budget.
3. Understand language model biases in robotics: The consistent bias hierarchy (Color > Object > ... > Verb > Size) is a valuable empirical finding. If you’re building on pretrained vision-language models for robotics, know that they likely have a strong visual bias (color, shape) and a weak grasp on verbs (actions) and relative attributes (size). You can preemptively design your evaluations and data to compensate for this.