VisionZip: Longer is Better but Not Necessary in Vision Language Models

TL;DR

VisionZip reduces redundancy in visual tokens for vision-language models, leading to faster inference and improved performance by selecting informative tokens, thus addressing inefficiencies caused by overly long visual representations.

Contribution

Introduction of VisionZip, a method that selects informative visual tokens to reduce redundancy and improve efficiency in vision-language models, applicable to image, video, and dialogue tasks.

Findings

VisionZip outperforms previous methods by at least 5% in performance.

Increases inference speed by up to 8x.

Enables faster inference for larger models with better results.

Abstract

Recent advancements in vision-language models have enhanced performance by increasing the length of visual tokens, making them much longer than text tokens and significantly raising computational costs. However, we observe that the visual tokens generated by popular vision encoders, such as CLIP and SigLIP, contain significant redundancy. To address this, we introduce VisionZip, a simple yet effective method that selects a set of informative tokens for input to the language model, reducing visual token redundancy and improving efficiency while maintaining model performance. The proposed VisionZip can be widely applied to image and video understanding tasks and is well-suited for multi-turn dialogues in real-world scenarios, where previous methods tend to underperform. Experimental results show that VisionZip outperforms the previous state-of-the-art method by at least 5% performance gains across nearly all settings. Moreover, our method significantly enhances model inference speed, improving the prefilling time by 8x and enabling the LLaVA-Next 13B model to infer faster than the LLaVA-Next 7B model while achieving better results. Furthermore, we analyze the causes of this redundancy and encourage the community to focus on extracting better visual features rather than merely increasing token length. Our code is available at https://github.com/dvlab-research/VisionZip .