Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuning

TL;DR

This paper introduces LLaVA-Meteor, a novel vision token projection method that compresses visual tokens efficiently, maintaining accuracy while significantly reducing computational costs in visual instruction tuning.

Contribution

The paper proposes a Top-Down Compression paradigm with a Flash Global Fusion module and a Visual-Native Selection mechanism to improve efficiency in vision-language models.

Findings

Reduces visual tokens by 75-95% with maintained or improved performance.

Achieves state-of-the-art results across 12 benchmarks.

Enhances vision modeling with local-to-single scanning and selective token assessment.

Abstract

Visual instruction tuning aims to enable large language models to comprehend the visual world, with a pivotal challenge lying in establishing an effective vision-to-language projection. However, existing methods often grapple with the intractable trade-off between accuracy and efficiency. In this paper, we present LLaVA-Meteor, a novel approach designed to break this deadlock, equipped with a novel Top-Down Compression paradigm that strategically compresses visual tokens without compromising core information. Specifically, we construct a trainable Flash Global Fusion module based on efficient selective state space operators, which aligns the feature space while enabling each token to perceive holistic visual context and instruction preference at low cost. Furthermore, a local-to-single scanning manner is employed to effectively capture local dependencies, thereby enhancing the model's capability in vision modeling. To alleviate computational overhead, we explore a Visual-Native Selection mechanism that independently assesses token significance by both the visual and native experts, followed by aggregation to retain the most critical subset. Extensive experiments show that our approach reduces visual tokens by 75--95% while achieving comparable or superior performance across 12 benchmarks, significantly improving efficiency.