SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs

TL;DR

This paper discovers that only a small subset of attention heads in multimodal large language models are responsible for visual understanding, and introduces SparseMM, a method to accelerate inference by leveraging this sparsity.

Contribution

The paper reveals the sparsity phenomenon in visual heads of MLLMs and proposes SparseMM, a KV-Cache optimization strategy that improves inference efficiency while preserving accuracy.

Findings

Sparse heads constitute less than 5% of attention heads in MLLMs.

SparseMM achieves 1.38x real-time acceleration during generation.

Memory usage is reduced by 52% without performance loss.

Abstract

Multimodal Large Language Models (MLLMs) are commonly derived by extending pre-trained Large Language Models (LLMs) with visual capabilities. In this work, we investigate how MLLMs process visual inputs by analyzing their attention mechanisms. We reveal a surprising sparsity phenomenon: only a small subset (approximately less than 5%) of attention heads in LLMs actively contribute to visual understanding, termed visual heads. To identify these heads efficiently, we design a training-free framework that quantifies head-level visual relevance through targeted response analysis. Building on this discovery, we introduce SparseMM, a KV-Cache optimization strategy that allocates asymmetric computation budgets to heads in LLMs based on their visual scores, leveraging the sparity of visual heads for accelerating the inference of MLLMs. Compared with prior KV-Cache acceleration methods that ignore the particularity of visual, SparseMM prioritizes stress and retaining visual semantics during decoding. Extensive evaluations across mainstream multimodal benchmarks demonstrate that SparseMM achieves superior accuracy-efficiency trade-offs. Notably, SparseMM delivers 1.38x real-time acceleration and 52% memory reduction during generation while maintaining performance parity on efficiency test. Our project is open sourced at https://github.com/CR400AF-A/SparseMM.