When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs

Paper: 2602.17659 Authors: Yu Fang, Yuchun Feng, Dong Jing, Jiaqi Liu, Yue Yang, Zhenyu Wei, Daniel Szafir, Mingyu Ding Categories: cs.CV, cs.RO

Abstract

Vision-Language-Action (VLA) models are designed to translate natural language instructions into robotic actions by grounding language in visual perception. However, this research reveals a critical failure mode: when instructions lack strong scene-specific supervision, VLAs fall back on visual shortcuts learned from dataset biases rather than following language intent. The authors introduce LIBERO-CF, the first counterfactual benchmark that systematically evaluates language following by presenting alternative instructions in visually similar scenarios. Their evaluation exposes widespread counterfactual failures across state-of-the-art VLAs. To address this, they propose Counterfactual Action Guidance (CAG), a dual-branch inference scheme that combines a standard VLA policy with a language-unconditioned Vision-Action module to explicitly regularize language conditioning. CAG achieves substantial improvements in both language following accuracy and task success without requiring additional training data or architectural modifications.

Key Contributions

• LIBERO-CF Benchmark: First systematic counterfactual evaluation framework for VLAs that tests language following capability by assigning alternative instructions under visually plausible layouts
• Counterfactual Failure Analysis: Comprehensive characterization of how VLAs rely on visual shortcuts and dataset biases, ignoring language instructions in under-observed scenarios
• Counterfactual Action Guidance (CAG): A plug-and-play dual-branch inference method that regularizes language conditioning through counterfactual comparison with vision-only policies
• Extensive Validation: Demonstrated 9.7% improvement in language following accuracy and up to 17.2% improvement in real-world task success across diverse VLA architectures

The Problem: Visual Shortcuts Override Language

VLAs are trained on large-scale robot manipulation datasets where certain object-action pairs appear frequently. This creates strong visual priors: when a robot sees a frequently manipulated object, it tends to interact with it regardless of what the language instruction actually says. The paper identifies this as counterfactual failure - the model fails when presented with counterfactual instructions that contradict its learned visual shortcuts.

For example, in a kitchen scene with both a red mug and a blue plate, if the training data predominantly shows “pick up the red mug,” the VLA may continue picking up the red mug even when instructed to “pick up the blue plate.” The visual presence of the familiar red mug overrides the language specification.

This phenomenon is particularly problematic because:

• Standard benchmarks don’t expose it (they use instructions aligned with training distribution)
• It undermines the core promise of language-conditioned control
• It limits generalization to novel task compositions

LIBERO-CF: A Counterfactual Benchmark

The authors construct LIBERO-CF by taking existing LIBERO manipulation tasks and systematically creating counterfactual variants. The key insight is to keep visual layouts similar while swapping instruction targets to objects or goals that are less frequently associated with the observed scene configuration.

The benchmark includes:

• Spatial counterfactuals: Same objects, different spatial relationships (“put A on B” vs “put B on A”)
• Object counterfactuals: Multiple similar objects, instruction specifies the less-common target
• Action counterfactuals: Same scene, different manipulation primitives

Each counterfactual pair allows direct measurement of whether the model follows language or defaults to visual priors. The evaluation metric $\pi_{0.5}$ measures the probability of selecting the correct action when both the factual and counterfactual options are equally plausible visually.

Counterfactual Action Guidance (CAG)

CAG addresses counterfactual failures through a simple but effective principle: compare what the model does with language against what it would do without language. If they’re too similar, the model isn’t really using the language instruction.

The method works as follows:

1. Dual-branch inference: Run both a VLA policy $\pi_{\text{VLA}}(a|v,l)$ and a Vision-Action policy $\pi_{\text{VA}}(a|v)$ that sees only visual input
2. Counterfactual comparison: Compute the divergence between their action distributions
3. Regularized action selection: Adjust the VLA’s action to increase divergence from the VA baseline

Mathematically, CAG optimizes:

$a^* = \arg\max_a \left[ \log \pi_{\text{VLA}}(a|v,l) - \lambda \log \pi_{\text{VA}}(a|v) \right]$

where $\lambda$ controls the strength of regularization. This formulation has an elegant interpretation: it encourages actions that the VLA assigns high probability to, but the VA model assigns low probability to - precisely the actions that require language understanding.

Two variants are proposed:

• CAG-free: Uses the pretrained VLA’s vision encoder as the VA module (training-free)
• CAG-trained: Trains a dedicated VA module on the same data without language labels

Experimental Results

The evaluation spans multiple dimensions:

LIBERO-CF Benchmark Performance:

• Baseline VLAs show 15-25% counterfactual failure rates
• CAG-free improves language following accuracy by 9.7% and task success by 3.6%
• CAG-trained achieves 15.5% and 8.5% improvements respectively
• Improvements are consistent across different VLA architectures (OpenVLA, Octo, RT-1)

Under-observed Task Generalization:

• On tasks with limited training examples, CAG shows even larger gains
• Reduces over-reliance on frequent object-action associations
• Maintains performance on well-observed tasks while improving on rare ones

Real-world Robot Experiments:

• Tested on physical manipulation tasks with natural language instructions
• 9.4% reduction in counterfactual failures
• 17.2% average improvement in task success rate
• Demonstrates practical applicability beyond simulation

Ablation Studies:

• The regularization strength $\lambda$ requires tuning but is robust across a range
• Both CAG variants outperform baseline, with trained VA showing best results
• The method is complementary to other improvements like better pretraining

Implications and Future Directions

This work reveals a fundamental tension in VLA design: vision provides strong priors that can override weaker language signals. While this might seem like a bug, it’s actually a rational response to training data statistics - if certain visual patterns reliably predict actions, why attend to potentially noisy language?

The CAG solution is philosophically interesting because it doesn’t try to make the VLA “use language more” directly. Instead, it uses counterfactual reasoning: by comparing against a vision-only baseline, it identifies when language should matter and amplifies those signals.

Several directions emerge:

• Curriculum learning: Gradually increase counterfactual difficulty during training
• Data augmentation: Systematically generate counterfactual training examples
• Architecture design: Build explicit language-vision competition mechanisms into model architecture
• Evaluation protocols: Adopt counterfactual testing as standard practice for VLA benchmarks

The work also raises questions about what “language grounding” really means. Is a model truly grounded if it only uses language when vision is ambiguous? Or should language always modulate behavior, even when vision provides strong cues?

Takeaways

1. VLAs suffer from systematic counterfactual failures where visual shortcuts override language instructions, especially for under-observed task compositions
2. LIBERO-CF provides the first systematic benchmark for evaluating language following capability through counterfactual instruction assignments
3. Counterfactual Action Guidance (CAG) offers a plug-and-play solution that improves language conditioning without retraining or architectural changes
4. The dual-branch design explicitly regularizes language use by comparing VLA actions against vision-only baselines during inference
5. Substantial improvements across benchmarks: 9.7-15.5% gains in language following accuracy and up to 17.2% in real-world task success
6. The method generalizes across VLA architectures and is particularly effective for under-observed tasks where visual shortcuts are most problematic
7. Counterfactual evaluation should become standard practice for assessing whether VLAs truly ground language or merely correlate it with visual patterns