The Expense of Seeing: Attaining Trustworthy Multimodal Reasoning

Paper: 2604.20665 Authors: Karan Goyal, Dikshant Kukreja (KDD) Categories: cs.AI, cs.CV

Abstract

The rapid proliferation of Vision-Language Models (VLMs) operates on a dangerous, unquestioned axiom: that current VLMs faithfully synthesise multimodal data. This paper argues they do not. Rather than extracting grounded knowledge from visual inputs, state-of-the-art models frequently exhibit functional blindness, exploiting strong language priors to bypass severe visual representation bottlenecks.

Key Problem: Functional Blindness

VLMs often:

• Rely on language priors rather than visual evidence
• Bypass visual representation bottlenecks via language priors
• Fail to truly “see” and integrate visual information

Proposed Framework: Modality Translation Protocol

Novel information-theoretic approach that translates semantic payloads rather than ablating them, formulating three metrics:

1. Toll of Seeing (ToS): Cost of relying on visual vs. language information
2. Curse of Seeing (CoS): Performance degradation when forced to use visual info
3. Fallacy of Seeing (FoS): Mismatch between claimed visual understanding and actual behavior

Semantic Sufficiency Criterion (SSC): Combined metric to evaluate multimodal trustworthiness

Divergence Law of Multimodal Scaling

As language engines scale to unprecedented reasoning capabilities, the mathematical penalty of the visual knowledge bottleneck paradoxically increases.

Takeaways

• Current VLM evaluation confounds dataset biases with architectural incapacity
• Need to shift from passive diagnostic to active architectural blueprint
• KDD community should abandon “multimodal gain” pursuit
• Trustworthy multimodal reasoning requires fundamentally different architecture