Object-Centric Vision Token Pruning for Vision Language Models

TL;DR

This paper introduces OC-VTP, a lightweight, direct method for selecting the most representative vision tokens in vision language models, significantly improving inference efficiency while maintaining accuracy.

Contribution

OC-VTP is a novel, guaranteed approach that requires only light pre-training and can be integrated into existing VLMs without fine-tuning, enhancing efficiency and interpretability.

Findings

Consistently preserves accuracy across various pruning ratios.

Requires only lightweight pre-training of a small pruner.

Demonstrates interpretability of selected tokens.

Abstract

In Vision Language Models (VLMs), vision tokens are quantity-heavy yet information-dispersed compared with language tokens, thus consume too much unnecessary computation. Pruning redundant vision tokens for high VLM inference efficiency has been continuously studied but all existing methods resort to indirect and non-guaranteed ways. We propose OC-VTP, a direct and guaranteed approach to select the most representative vision tokens for high-efficiency yet accuracy-preserving VLM inference. Our OC-VTP requires merely light-weight pre-training of a small object-centric vision token pruner, which can then be inserted into existing VLMs, without fine-tuning of any models on any datasets. It is gauranteed that the most representative vision tokens are kept by minimizing the error in reconstructing the original unpruned tokens from the selected ones. Across any vision pruning ratios, i.e., inference efficiency, our OC-VTP consistently helps mainstream VLMs to preserve the highest inference accuracy. Our pruning also demonstrates interesting interpretability. Our codes are available at https://github.com/GarryLarry010131/OC-VTP.