Information Router for Mitigating Modality Dominance in Vision-Language Models

TL;DR

This paper introduces MoIR, a multi-modal information routing method that explicitly balances modality contributions by enriching token representations, improving robustness and performance in vision-language models under modality imbalance.

Contribution

MoIR is a novel information-level fusion technique that reduces modality disparity before fusion, enabling better handling of degraded or unbalanced modalities in vision-language models.

Findings

MoIR achieves more balanced modality contributions across benchmarks.

It improves robustness and downstream performance, especially with modality degradation.

Experimental results confirm the effectiveness of explicit information modification in multi-modal reasoning.

Abstract

Vision Language models (VLMs) have demonstrated strong performance across a wide range of benchmarks, yet they often suffer from modality dominance, where predictions rely disproportionately on a single modality. Prior approaches primarily address this issue by steering model's attention allocation, implicitly assuming that all modalities provide sufficient information. However, attention only determines where the model focuses, and cannot enrich information that is missing or ambiguous. In the real world, input modalities often differ in information density and their signal-to-noise ratios. In such cases, simply adjusting model's attention does not resolve the underlying lack of information. In this paper, we propose \textsc{MoIR}: \textit{Multi-modal Information Router}, an information-level fusion method that explicitly reduces information disparity prior to fusion. \textsc{MoIR} identifies less informative tokens and routes complementary information from a stronger modality, constructing information-dense token representations before they are processed by a large language model. By modifying information availability, \textsc{MoIR} enables reliable shifts in modality dominance, even when one modality is degraded. We evaluate \textsc{MoIR} on three widely used multi-modal benchmarks across multiple model backbones. Experimental results show that \textsc{MoIR} consistently demonstrates more balanced modality contribution, and improves robustness and downstream performance, particularly even under modality degradation. These findings demonstrate that explicitly modifying cross-modal information is an effective and complementary strategy for mitigating modality dominance in multi-modal reasoning models.