Sparse Shortcuts: Facilitating Efficient Fusion in Multimodal Large Language Models

TL;DR

This paper introduces SparseCut, a novel architecture with sparse shortcut connections for multimodal large language models, enabling efficient hierarchical visual feature fusion that improves performance without added computational costs.

Contribution

SparseCut provides a new cross-modal fusion method with sparse shortcuts and multi-grained feature fusion, enhancing semantic integration in MLLMs efficiently and scalably.

Findings

Significantly improves MLLM performance on multiple benchmarks.

Enables hierarchical visual feature fusion without increasing computational overhead.

Demonstrates generality across different base LLMs.

Abstract

With the remarkable success of large language models (LLMs) in natural language understanding and generation, multimodal large language models (MLLMs) have rapidly advanced in their ability to process data across multiple modalities. While most existing efforts focus on scaling up language models or constructing higher-quality training data, limited attention has been paid to effectively integrating cross-modal knowledge into the language space. In vision-language models, for instance, aligning modalities using only high-level visual features often discards the rich semantic information present in mid- and low-level features, limiting the model's ability of cross-modality understanding. To address this issue, we propose SparseCut, a general cross-modal fusion architecture for MLLMs, introducing sparse shortcut connections between the cross-modal encoder and the LLM. These shortcut connections enable the efficient and hierarchical integration of visual features at multiple levels, facilitating richer semantic fusion without increasing computational overhead. We further introduce an efficient multi-grained feature fusion module, which performs the fusion of visual features before routing them through the shortcuts. This preserves the original language context and does not increase the overall input length, thereby avoiding an increase in computational complexity for the LLM. Experiments demonstrate that SparseCut significantly enhances the performance of MLLMs across various multimodal benchmarks with generality and scalability for different base LLMs.