CoViPAL: Layer-wise Contextualized Visual Token Pruning for Large Vision-Language Models

TL;DR

CoViPAL introduces a layer-wise, context-aware token pruning method for large vision-language models, significantly reducing computational costs while maintaining high accuracy through a lightweight, model-agnostic module.

Contribution

It proposes a novel Plug-and-Play Pruning Module that effectively prunes redundant visual tokens across layers, outperforming existing methods without additional training.

Findings

Outperforms training-free pruning methods at the same token budget

Surpasses training-based methods with similar supervision

Enhances inference efficiency without accuracy loss

Abstract

Large Vision-Language Models (LVLMs) process multimodal inputs consisting of text tokens and vision tokens extracted from images or videos. Due to the rich visual information, a single image can generate thousands of vision tokens, leading to high computational costs during the prefilling stage and significant memory overhead during decoding. Existing methods attempt to prune redundant vision tokens, revealing substantial redundancy in visual representations. However, these methods often struggle in shallow layers due to the lack of sufficient contextual information. We argue that many visual tokens are inherently redundant even in shallow layers and can be safely and effectively pruned with appropriate contextual signals. In this work, we propose CoViPAL, a layer-wise contextualized visual token pruning method that employs a Plug-and-Play Pruning Module (PPM) to predict and remove redundant vision tokens before they are processed by the LVLM. The PPM is lightweight, model-agnostic, and operates independently of the LVLM architecture, ensuring seamless integration with various models. Extensive experiments on multiple benchmarks demonstrate that CoViPAL outperforms training-free pruning methods under equal token budgets and surpasses training-based methods with comparable supervision. CoViPAL offers a scalable and efficient solution to improve inference efficiency in LVLMs without compromising accuracy.