Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Acceleration

TL;DR

This paper critically examines visual token pruning in vision-language models, revealing its limitations on localization tasks and proposing FEATHER, a multistage pruning method that significantly improves performance on vision-centric benchmarks.

Contribution

The paper identifies a core flaw in early token pruning strategies and introduces FEATHER, a simple yet effective multistage pruning approach that enhances visual token preservation.

Findings

FEATHER achieves over 5x performance improvement on localization benchmarks.

Early pruning strategies often discard crucial visual tokens, impairing fine-grained visual tasks.

Benchmarks may not fully assess the visual capabilities of accelerated models.

Abstract

Recent works on accelerating Vision-Language Models achieve strong performance across a variety of vision-language tasks despite highly compressing visual information. In this work, we examine the popular acceleration approach of early pruning of visual tokens inside the language model. Surprisingly, we find that while strong performance is maintained across many tasks, it exhibits drastically different behavior for a subset of vision-centric tasks such as localization. Upon further investigation, we uncover a core issue with the acceleration approach where most tokens towards the top of the image are pruned away. Yet, on many benchmarks aiming to evaluate vision-centric capabilities, strong performance persists with the flawed pruning strategy, highlighting these benchmarks' limited ability to assess fine-grained visual capabilities. Based on these findings, we propose FEATHER (Fast and Effective Acceleration wiTH Ensemble cRiteria), a straightforward approach that resolves the discovered early-layer pruning issue and further enhances the preservation of relevant tokens via multistage pruning with early uniform sampling to ensure broad image coverage. With comparable computational savings, we find that FEATHER achieves more than 5x performance improvement on the vision-centric localization benchmarks compared to the original acceleration approach.