Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders

TL;DR

This paper reveals that in multimodal large language models, multiple vision encoders often provide redundant information, and selectively masking encoders can improve performance and efficiency, challenging the assumption that more encoders always enhance results.

Contribution

The study introduces two metrics, CUR and IG, to quantify encoder utility and redundancy, providing new insights into encoder specialization and redundancy in multimodal models.

Findings

Encoders show high redundancy on general tasks, allowing for simpler models.

Masking certain encoders can improve task performance and reduce resources.

Some encoders negatively impact performance, indicating detrimental effects.

Abstract

Recent multimodal large language models (MLLMs) increasingly integrate multiple vision encoders to improve performance on various benchmarks, assuming that diverse pretraining objectives yield complementary visual signals. However, we show this assumption often fails in practice. Through systematic encoder masking across representative multi encoder MLLMs, we find that performance typically degrades gracefully, and sometimes even improves, when selected encoders are masked, revealing pervasive encoder redundancy. To quantify this effect, we introduce two principled metrics: the Conditional Utilization Rate (CUR), which measures an encoder s marginal contribution in the presence of others, and the Information Gap (IG), which captures heterogeneity in encoder utility within a model. Using these tools, we observe: (i) strong specialization on tasks like OCR and Chart, where a single encoder can dominate with a CUR greater than 90 percent, (ii) high redundancy on general VQA and knowledge based tasks, where encoders are largely interchangeable, (iii) instances of detrimental encoders with negative CUR. Notably, masking specific encoders can yield up to 16 percent higher accuracy on a specific task category and 3.6 percent overall performance boost compared to the full model.Furthermore, single and dual encoder variants recover over 90 percent of baseline on most non OCR tasks with substantially lower training resources and inference latency. Our analysis challenges the more encoders are better heuristic in MLLMs and provides actionable diagnostics for developing more efficient and effective multimodal architectures.