SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference

TL;DR

SparseVLM introduces a training-free, text-guided method for visual token sparsification in vision-language models, significantly reducing computational costs while maintaining high accuracy in image and video understanding tasks.

Contribution

It proposes a novel, training-free token pruning strategy guided by text, with adaptive sparsification and token recycling, improving efficiency of VLMs without extra training.

Findings

Achieves 54% reduction in FLOPs on LLaVA

Decreases CUDA latency by 37%

Maintains 97% of original accuracy

Abstract

In vision-language models (VLMs), visual tokens usually bear a significant amount of computational overhead despite sparsity of information in them when compared to text tokens. To address this, most existing methods learn a network to prune redundant visual tokens using certain training data. Differently, we propose a text-guided training-free token optimization mechanism dubbed SparseVLM that eliminates the need of extra parameters or fine-tuning costs. Given that visual tokens complement text tokens in VLM's linguistic reasoning, we select relevant text tokens to rate the significance of visual tokens using self-attention matrices and, then, prune visual tokens using the proposed strategy to maximize sparsity while retaining information. In particular, we introduce a rank-based strategy to adaptively determine the sparsification ratio for each layer, alongside a token recycling method that compresses pruned tokens into more compact representations. Experimental results show that SparseVLM increases the efficiency of various VLMs in a number of image and video understanding tasks. For example, LLaVA when equipped with SparseVLM achieves 54% reduction in FLOPs, 37% decrease in CUDA latency while maintaining 97% of its original accuracy. Our code is available at https://github.com/Gumpest/SparseVLMs.