ZSPAPrune: Zero-Shot Prompt-Aware Token Pruning for Vision-Language Models

TL;DR

ZSPAPrune introduces a zero-shot, prompt-aware token pruning method for vision-language models that effectively reduces computational costs while maintaining high accuracy by balancing task relevance and diversity.

Contribution

It presents a novel prompt-aware token pruning approach that explicitly models task relevance, outperforming existing methods in efficiency and accuracy preservation.

Findings

Achieves up to 90% token pruning with minimal accuracy loss.

Reduces GPU memory and inference latency significantly.

Matches or surpasses state-of-the-art performance on multiple benchmarks.

Abstract

As the capabilities of Vision-Language Models (VLMs) advance, they can process increasingly large inputs, which, unlike in LLMs, generates significant visual token redundancy and leads to prohibitive inference costs. While many methods aim to reduce these costs by pruning visual tokens, existing approaches, whether based on attention or diversity, typically neglect the guidance of the text prompt and thus fail to prioritize task relevance. In this work, we propose a novel, zero-shot method that reframes the problem by introducing a prompt-aware perspective, explicitly modeling visual token pruning as a balance between task relevance and information diversity. Our hierarchical approach first selects a core set of task-relevant visual tokens and then supplements them with diversity tokens to preserve broader context. Experiments across multiple models and benchmarks show that our method achieves performance that matches or surpasses the state-of-the-art with only minimal accuracy loss, even when pruning up to 90\% of the tokens. Furthermore, these gains are accompanied by significant reductions in GPU memory footprint and inference latency.