Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment

TL;DR

This paper introduces VisionDrop, a training-free visual token pruning method for LVLMs that improves efficiency by selecting informative visual tokens based solely on visual attention, addressing cross-modal misalignment issues.

Contribution

The paper proposes a novel, training-free visual-only pruning framework that enhances token reduction in LVLMs without relying on textual signals or additional training.

Findings

VisionDrop achieves a 2.7x reduction in inference latency.

It reduces FLOPs by 6x while maintaining 95.71% of original performance.

The method outperforms existing token reduction approaches across benchmarks.

Abstract

Large Vision-Language Models (LVLMs) encode visual inputs as dense sequences of patch-level tokens to capture fine-grained semantics. These visual tokens often outnumber their textual counterparts by a large margin, leading to substantial computational overhead and limiting the scalability of LVLMs in practice. Previous efforts have explored visual token reduction either prior to or within the large language models (LLMs). However, most in-LLM reduction approaches rely on text-conditioned interactions, implicitly assuming that textual tokens can reliably capture the importance of visual tokens. In this work, we revisit this assumption and reveal causal, semantic, and spatial forms of cross-modal misalignment. These misalignments undermine the effectiveness of text-guided visual token reduction. To address this, we introduce VisionDrop, a training-free, visual-only pruning framework that selects informative visual tokens based on intra-modal (visual-to-visual) attention, without relying on textual signals. To further suppress redundancy throughout the model hierarchy, we treat the visual encoder and the LLM as a unified system and design a progressive pruning pipeline. Our method performs dominant token selection and lightweight contextual merging at multiple stages, enabling fine-grained visual information to be retained even under aggressive token budgets. Extensive experiments across diverse benchmarks show that VisionDrop achieves consistent improvements over existing approaches, despite requiring no additional training or complex modifications. Notably, when integrated with LLaVA-NeXT-7B, VisionDrop achieves a 2.7x reduction in inference latency and 6x in FLOPs, while retaining 95.71% of the original performance.