Scale Can't Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning

Paper: 2602.23351 Authors: Amita Kamath, Jack Hessel, Khyathi Chandu, Jena D. Hwang, Kai-Wei Chang, Ranjay Krishna Categories: cs.CL, cs.CV

Abstract

Vision-Language Models (VLMs) continue to struggle with reasoning tasks despite massive scaling efforts. This research identifies reporting bias as the root cause: natural human communication about visual content systematically omits tacit information essential for certain types of reasoning. For instance, people naturally say “at the game today!” rather than “a photo of 37 people standing behind a field.” The authors analyze training data from OpenCLIP, LLaVA-1.5, and Molmo through pragmatic theory, finding that four critical reasoning skills—spatial, temporal, negation, and counting—are severely underrepresented. Their experiments demonstrate that scaling alone cannot solve this problem, but targeted annotation of tacit information can effectively address these gaps.

Key Contributions

• Identifies reporting bias as a fundamental limitation in VLM training data that prevents emergence of reasoning capabilities
• Provides systematic analysis of web-scale and synthetic training corpora through the lens of pragmatic theory
• Demonstrates that spatial, temporal, negation, and counting reasoning are systematically suppressed in natural image captions
• Shows empirically that scaling data size, model parameters, or multilingual coverage does not overcome reporting bias
• Proves that targeted annotation strategies focusing on tacit information can effectively improve reasoning performance

The Pragmatics of Reporting Bias

The paper grounds its analysis in Grice’s Cooperative Principle and conversational maxims from pragmatics. When humans communicate about images, they follow implicit rules: be informative but not overly detailed, be relevant to the conversational context, and be brief. This creates a systematic gap between what is visible in an image and what people naturally describe.

For example, when captioning a sports event photo, a person might write “amazing game!” which conveys emotional content and context but omits explicit spatial arrangements, exact counts of people, temporal sequences, or negative information (what is not present). This reporting bias is not random noise—it’s a structured pattern driven by how human communication works.

The researchers formalize this through four reasoning categories that are particularly affected:

• Spatial reasoning: Descriptions rarely specify precise spatial relationships (“the cup is to the left of the plate”)
• Temporal reasoning: Captions seldom indicate sequences or time-based relationships
• Negation reasoning: What is absent from an image is almost never mentioned
• Counting reasoning: Exact quantities are typically omitted unless contextually critical

Empirical Analysis of Training Data

The authors conducted extensive analysis of three major VLM training datasets:

OpenCLIP (trained on LAION-2B): A web-scale dataset of image-text pairs scraped from the internet. Analysis revealed that only 2.3% of captions contain spatial prepositions, 0.8% contain temporal markers, 1.1% contain negations, and 3.7% contain numbers.

LLaVA-1.5: Uses synthetic captions generated by GPT-4V. Despite being machine-generated with explicit instructions for detail, the synthetic captions still exhibit reporting bias, though less severe than web data. This suggests that even language models trained on human text inherit and reproduce pragmatic patterns.

Molmo: Combines web data with human annotations. Shows slightly better coverage of reasoning-relevant information but still falls short of what’s needed for robust reasoning.

The analysis employed both lexical pattern matching and semantic classification to identify reasoning-relevant content, ensuring robust measurement across different expression styles.

Experimental Results: Scale Doesn’t Help

The paper presents three key experimental findings:

Finding 1: VLMs fail on reasoning tasks suppressed by reporting bias

Using curated benchmarks for spatial (VSR), temporal (STAR), negation (VALSE), and counting (CountBench) reasoning, the authors tested OpenCLIP, LLaVA-1.5, and Molmo. Performance was consistently poor across all models, with accuracy often near random chance for negation and counting tasks.

Finding 2: Scaling doesn’t induce reasoning emergence

The researchers tested whether increasing model size (from 400M to 7B parameters), data size (from 400M to 2B image-text pairs), or language diversity (English-only vs. multilingual) would improve reasoning. Results showed minimal improvements, with some configurations even degrading performance. This directly contradicts the “scaling hypothesis” that sufficient scale leads to emergent capabilities.

Finding 3: Targeted annotation works

When models were fine-tuned on small datasets (10K-100K examples) specifically annotated to include tacit information about spatial relationships, temporal sequences, negations, and counts, performance improved dramatically—often by 20-40 percentage points. This demonstrates that the issue is data quality, not model capacity.

Implications for VLM Development

This research has profound implications for how we approach VLM training:

Rethinking data curation: The field has largely assumed that web-scale data collection is sufficient. This work shows that scale without intentional curation of reasoning-relevant information is insufficient. We need annotation protocols that explicitly capture tacit information.

Beyond synthetic data: While synthetic caption generation (e.g., using GPT-4V) can help, it’s not a complete solution since language models inherit reporting bias from their training data. Synthetic data generation must be guided by explicit reasoning requirements.

The limits of emergence: The finding that scaling doesn’t induce reasoning capabilities challenges a core assumption in modern AI research. It suggests that some capabilities require explicit supervision rather than emerging from scale alone.

Practical annotation strategies: The success of targeted annotation suggests a path forward: identify specific reasoning gaps, design annotation protocols to capture relevant tacit information, and augment training data strategically rather than indiscriminately.

Takeaways

1. Reporting bias—the systematic omission of tacit information in natural communication—fundamentally limits VLM reasoning capabilities in spatial, temporal, negation, and counting domains.

2. Scaling model size, data volume, or language diversity does not overcome reporting bias; these reasoning gaps persist even in models trained on billions of image-text pairs.

3. Web-scale datasets and even synthetic captions generated by large language models exhibit reporting bias, as they reflect natural human communication patterns.

4. Targeted annotation strategies that explicitly capture tacit information can effectively address reasoning gaps, achieving substantial performance improvements with relatively small amounts of curated data.

5. The field needs to move beyond scale-focused approaches toward intentional data curation methods that account for pragmatic aspects of human communication.