Rethinking the Mixture of Vision Encoders Paradigm for Enhanced Visual Understanding in Multimodal LLMs

TL;DR

This paper systematically studies fusion strategies for Mixture of Vision Encoders in multimodal large language models, proposing a lightweight architecture called LEO that improves performance and generalizes well across diverse vision-language tasks.

Contribution

It introduces a set of principles for effective token-level fusion in MoVE-based MLLMs and presents LEO, a simple architecture that outperforms existing methods on multiple benchmarks.

Findings

LEO achieves superior results on most vision-language benchmarks.

LEO generalizes well to autonomous driving domain without architectural changes.

The proposed fusion principles enhance multimodal understanding in MLLMs.

Abstract

Mixture of Vision Encoders (MoVE) has emerged as a powerful approach to enhance the fine-grained visual understanding of multimodal large language models (MLLMs), improving their ability to handle tasks such as complex optical character recognition and scene understanding. Despite these advances, effectively combining diverse encoders and their visual tokens, while also scaling to high-resolution inputs, remains an open challenge. In this work, we conduct a systematic study of fusion designs for MoVE-based MLLMs, highlighting principles for token-level integration across complementary encoders. Our study shows that a lightweight recipe consisting of post-adaptation fusion with independent projectors, tile-level sequence interleaving, and dynamic tiling with global context delivers strong performance on diverse benchmarks. We integrate these principles into a simple and effective architecture that we call LEO. Extensive evaluation on 11 vision-language benchmarks demonstrates that LEO achieves better results on the majority of tasks compared to existing MoVE-based approaches. Furthermore, LEO adapts effectively to the specialized domain of autonomous driving without altering its architecture or training recipe, achieving competitive performance against established baselines and thereby highlighting its ability to generalize. The code is available at https://github.com/Mozhgan91/LEO.