To Preserve or To Compress: An In-Depth Study of Connector Selection in Multimodal Large Language Models

TL;DR

This study systematically evaluates how different connector types in multimodal large language models affect performance across various perception and reasoning tasks, providing guidance for architecture design.

Contribution

It introduces a unified classification of connectors and benchmarks their impact on diverse perception and reasoning tasks in MLLMs.

Findings

Feature-preserving connectors excel in fine-grained perception tasks.

Feature-compressing connectors offer speed advantages and perform well in coarse perception and reasoning.

Insights guide better MLLM architecture design.

Abstract

In recent years, multimodal large language models (MLLMs) have garnered significant attention from both industry and academia. However, there is still considerable debate on constructing MLLM architectures, particularly regarding the selection of appropriate connectors for perception tasks of varying granularities. This paper systematically investigates the impact of connectors on MLLM performance. Specifically, we classify connectors into feature-preserving and feature-compressing types. Utilizing a unified classification standard, we categorize sub-tasks from three comprehensive benchmarks, MMBench, MME, and SEED-Bench, into three task types: coarse-grained perception, fine-grained perception, and reasoning, and evaluate the performance. Our findings reveal that feature-preserving connectors excel in \emph{fine-grained perception} tasks due to their ability to retain detailed visual information. In contrast, feature-compressing connectors, while less effective in fine-grained perception tasks, offer significant speed advantages and perform comparably in \emph{coarse-grained perception} and \emph{reasoning} tasks. These insights are crucial for guiding MLLM architecture design and advancing the optimization of MLLM architectures.