Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts

Paper: 2604.08541 Authors: Haolei Xu, Haiwen Hong, Hongxing Li, Rui Zhou, Yang Zhang, Longtao Huang, Hui Xue, Yongliang Shen, Weiming Lu, Yueting Zhuang Categories: cs.CV

The Gap

Multimodal Mixture-of-Experts models have achieved impressive performance on vision-language tasks, but there is a strange failure mode hiding beneath the surface. Present a grade-school math problem as an image, and the model extracts all the numbers and text perfectly—yet produces the wrong answer due to reasoning errors. Give the exact same problem as pure text, and it solves it correctly with ease.

The authors call this Seeing but Not Thinking: the model perceives visual content accurately but fails to reason correctly, despite possessing the requisite capability on semantically equivalent text inputs.

This is not a niche edge case. On controlled experiments using MATH500 rendered as high-resolution images, 68.2% to 73.1% of failures stem from reasoning errors rather than perception errors. The model sees correctly but thinks incorrectly when visual inputs are involved.

The natural first hypothesis is cross-modal semantic alignment failure: maybe visual information, though correctly perceived, fails to align with the textual semantic space at the representation level. But the authors rule this out through concept intervention experiments, demonstrating that MoE architectures do exhibit cross-modal semantic sharing in middle layers, just like dense models.

If semantic alignment is not the bottleneck, what is? The authors turn their attention to the routing mechanism—the core component distinguishing MoE from dense models. Through systematic analysis, they uncover two critical findings:

1. Layer-wise expert separation: Visual experts (handling image encoding) concentrate in early and terminal layers, while domain-specific reasoning experts cluster in middle layers.

2. Routing divergence: Image inputs induce significant routing divergence from text inputs precisely in these middle layers where domain experts concentrate. Crucially, greater routing divergence correlates with lower reasoning accuracy.

This leads to the Routing Distraction hypothesis: when processing visual inputs, the routing mechanism fails to adequately activate task-relevant reasoning experts in middle layers, instead directing computation toward less suitable experts. The visual modality does not impair intrinsic reasoning capabilities; rather, it causes suboptimal expert selection.

[Problem]
   |
   v
[Models perceive correctly
 but reason incorrectly
 on visual inputs]
   |
   v
[Hypothesis 1: Semantic alignment failure?]
   |
   v
[Test: Cross-modal concept intervention]
   |
   v
[Result: Semantic sharing exists in middle layers]
   |
   v
[Hypothesis 2: Routing distraction?]
   |
   v
[Analysis: Visual and domain experts
 are layer-wise separated
 Image inputs cause routing divergence
 in middle layers]
   |
   v
[Validation: Routing-guided intervention
 enhances domain expert activation
 Improves reasoning up to 3.17%]

The Increment

One sentence: Before this paper, the failure of multimodal MoE models on visual reasoning was attributed vaguely to “multimodal integration issues”; after it, we have a mechanistic explanation—routing distraction in middle layers—and a practical intervention method.

Core Mechanism

The paper proceeds in three stages: diagnosis, analysis, and intervention.

Stage 1: Verifying Cross-Modal Semantic Sharing

The authors design a concept intervention experiment. They construct an arithmetic completion task where the input consists of a digit image followed by textual arithmetic (e.g., image of “3” followed by ”+ 2 =”). They extract hidden state vectors for source digit S and target digit T from pure text inputs, then perform this intervention on image token hidden states:

ĥ_img^(l) ← h_img^(l) - α·h_src^(l) + α·h_tgt^(l)

This removes the source concept’s semantic vector from the image representation while adding the target concept’s vector. If the model’s output changes to match the target digit, the intervention succeeds.

Results show a clear inverted U-shaped pattern across layers: low success rates in early layers (visual features not yet aligned), high success rates exceeding 90% in middle layers (8-42), and sharp decline in terminal layers (output distribution already determined). This confirms that MoE architectures exhibit cross-modal semantic sharing, ruling out alignment failure as the sole explanation.

Stage 2: Identifying Layer-Wise Expert Specialization

The authors identify domain experts by comparing activation frequencies between domain-specific data (GSM8K for math) and general data (Alpaca). Experts with significantly higher activation on domain data are classified as domain experts. Similarly, they identify visual experts by comparing activation frequencies between image-rendered and text versions of the same content.

The key finding: visual experts and domain experts exhibit layer-wise separation. Visual experts cluster in early layers (initial visual encoding) and terminal layers (preparing outputs), while math experts concentrate in middle layers (6-42) where cross-modal semantic sharing occurs.

Stage 3: Analyzing Routing Divergence

The authors measure routing divergence between semantically equivalent text and image inputs using Jensen-Shannon Divergence (JSD) on expert activation distributions. They test three image versions with increasing visual complexity.

Results reveal that routing divergence exhibits a U-shaped distribution: larger in early/terminal layers (expected due to visual encoding and output preparation), but critically, the three image versions diverge primarily in middle layers (6-42) where domain experts concentrate. Greater routing divergence in middle layers correlates with lower reasoning accuracy.

The Intervention: Routing-Guided Enhancement

Based on the Routing Distraction hypothesis, the authors propose a simple intervention: explicitly enhance routing weights of identified domain experts during inference. They test two strategies:

• Soft intervention: Apply moderate additive enhancement to domain expert logits: r'_k ← r_k + λ·s(r) where s(r) is the standard deviation of expert logits.

• Hard intervention: Force domain expert logits to layer maximum: r'_k ← max_j(r_j) + δ.

The soft intervention achieves consistent improvements across three MoE models (Kimi-VL-16B, Qwen3-VL-30B, Llama4-Scout-109B) and six benchmarks, with gains up to 3.17% on complex visual reasoning tasks. Hard intervention shows mixed results, sometimes causing severe degradation on models with Top-1 routing.

[Cross-modal concept intervention]
         |
         v
[Semantic sharing verified
 in middle layers]
         |
         v
[Domain expert identification]
[Compare activation frequencies
 on domain vs general data]
         |
         v
[Visual expert identification]
[Compare activation frequencies
 on image vs text data]
         |
         v
[Finding: Layer-wise separation
 Visual experts: early + terminal
 Domain experts: middle layers]
         |
         v
[Routing divergence analysis]
[Image inputs cause divergence
 in middle layers]
         |
         v
[Routing-guided intervention]
[Enhance domain expert activation
 during inference]
         |
         v
[Consistent improvements
 across models and tasks]

Key Concepts

• Mixture-of-Experts (MoE): An architecture that activates only a sparse subset of experts for each input, enabling efficient scaling. The routing mechanism decides which experts to activate. Unlike dense models where all parameters process every input, MoE models route different inputs to different expert subsets, allowing specialization.

• Routing distraction: The phenomenon where visual inputs cause the routing mechanism to inadequately activate task-relevant reasoning experts, instead directing computation toward less suitable experts. This is not a failure of reasoning capability itself, but a failure of expert selection.

• Cross-modal semantic sharing: The property that visual and textual representations of the same concept converge to similar semantic spaces in middle layers of the model. This allows the model to process semantically equivalent information regardless of input modality.

• Layer-wise expert separation: The spatial distribution pattern where different types of experts concentrate in different layers. Visual experts handle modality-specific processing in early/terminal layers, while domain experts handle abstract reasoning in middle layers where semantic sharing occurs.

Framework Shift

Before (implicit assumption):        After (this paper):

[Multimodal MoE failures]            [Multimodal MoE failures]
          |                                    |
          v                                    v
[Vague attribution to                [Mechanistic diagnosis]
 "integration issues"]                        |
          |                                    v
          v                          [Cross-modal semantic sharing
[No clear mechanism]                  exists in middle layers]
          |                                    |
          v                                    v
[No targeted intervention]           [Visual and domain experts
                                      are layer-wise separated]
                                               |
                                               v
                                     [Routing divergence in
                                      middle layers causes
                                      inadequate domain expert
                                      activation]
                                               |
                                               v
                                     [Routing-guided intervention
                                      recovers reasoning performance]

From vague attribution to mechanistic understanding and targeted intervention, the core shift is treating routing behavior as a first-class object of analysis in multimodal reasoning failures.

Expert Assessment

Problem choice: This is a real and important phenomenon. As multimodal MoE models become the dominant paradigm for scaling vision-language systems, understanding their failure modes is critical. The Seeing but Not Thinking phenomenon is not just academically interesting—it has direct implications for deployment in applications requiring visual reasoning.

Method maturity: The diagnostic methodology is rigorous and well-designed. The cross-modal concept intervention experiment provides clean evidence for semantic sharing. The layer-wise expert analysis reveals structural patterns that were previously hidden. The routing divergence analysis establishes correlation with reasoning performance. The intervention validation demonstrates practical utility.

The main limitation is that the intervention requires task-specific configuration: identifying domain experts, determining optimal intervention layers and strengths. The paper acknowledges this and suggests that developing adaptive methods would improve practical applicability.

Experimental integrity: The experiments are carefully controlled. Using semantically equivalent text and image versions isolates reasoning from perception. Error analysis confirms that 68-73% of failures are reasoning errors, not perception errors. The evaluation spans three models with different scales and routing mechanisms (Top-K vs Top-1), demonstrating generality. The ablations examine intervention layer range, enhancement strength, and domain specificity.

One strength is the analysis of expert identification robustness: the paper shows that domain experts transfer effectively across tasks with different information structures, as long as the reference elicits target domain reasoning patterns. This suggests expert identification locates cognitive functions rather than sample-specific solutions.

Writing quality: The paper is well-structured and clearly written. The progression from diagnosis to analysis to intervention is logical. The figures effectively illustrate key findings. The related work section properly positions the contribution relative to prior work on semantic sharing, expert specialization, and routing intervention.

The limitation section is honest about scope: routing guidance cannot address perceptual errors, requires task-specific configuration, and the cross-modal semantic sharing experiments only verify alignment for simple numerical concepts.

Verdict: accept — This paper makes a solid mechanistic contribution to understanding multimodal MoE failures, provides rigorous experimental validation, and demonstrates practical utility through routing-guided intervention. The Routing Distraction hypothesis is well-supported and the intervention method is simple yet effective.

Takeaways

What practitioners can take from this work:

• Routing behavior matters for multimodal reasoning. In MoE architectures, it is not enough for the model to have the right experts—the routing mechanism must activate them appropriately. Visual inputs can cause routing distraction even when semantic alignment is preserved.

• Layer-wise expert analysis reveals hidden structure. Different types of experts concentrate in different layers. Understanding this spatial distribution can guide architecture design and intervention strategies.

• Inference-time routing intervention is a viable approach. You do not need to retrain the model to mitigate routing distraction. Explicitly enhancing domain expert activation during inference can recover reasoning performance.

• Domain expert identification is robust to information completeness. As long as the text reference elicits target domain reasoning patterns, identified experts transfer effectively to visual tasks with different information structures. This suggests experts encode cognitive functions rather than sample-specific solutions.

• Soft intervention outperforms hard intervention. Moderate additive enhancement preserves routing flexibility while systematically increasing domain expert activation probability. Forcing expert logits can disrupt the routing mechanism, especially in Top-1 routing architectures.

• The gap between perception and reasoning is real. Models can perceive correctly yet reason incorrectly. This is not just a multimodal issue—it reflects a deeper challenge in how neural networks integrate perception with abstract reasoning.

For researchers, this work opens several directions: Can we design routing mechanisms that are less susceptible to modality-induced distraction? Can we develop training objectives that encourage routers to allocate experts based on task requirements rather than input modality? Can we extend this analysis to other types of expert specialization beyond domain-specific reasoning?