Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts

TL;DR

This paper investigates why multimodal MoE models perceive images well but struggle with reasoning, revealing routing distractions as a key factor and proposing interventions that improve performance on visual reasoning tasks.

Contribution

The paper introduces the Routing Distraction hypothesis, analyzes layer-wise routing divergence, and proposes a routing-guided intervention to enhance reasoning in multimodal MoE models.

Findings

Routing divergence occurs in middle layers during visual input processing.

Routing-guided intervention improves reasoning performance by up to 3.17%.

Domain experts encode cognitive functions transferable across tasks.

Abstract

Multimodal Mixture-of-Experts (MoE) models have achieved remarkable performance on vision-language tasks. However, we identify a puzzling phenomenon termed Seeing but Not Thinking: models accurately perceive image content yet fail in subsequent reasoning, while correctly solving identical problems presented as pure text. Through systematic analysis, we first verify that cross-modal semantic sharing exists in MoE architectures, ruling out semantic alignment failure as the sole explanation. We then reveal that visual experts and domain experts exhibit layer-wise separation, with image inputs inducing significant routing divergence from text inputs in middle layers where domain experts concentrate. Based on these findings, we propose the Routing Distraction hypothesis: when processing visual inputs, the routing mechanism fails to adequately activate task-relevant reasoning experts. To validate this hypothesis, we design a routing-guided intervention method that enhances domain expert activation. Experiments on three multimodal MoE models across six benchmarks demonstrate consistent improvements, with gains of up to 3.17% on complex visual reasoning tasks. Our analysis further reveals that domain expert identification locates cognitive functions rather than sample-specific solutions, enabling effective transfer across tasks with different information structures.