Patch Ranking: Efficient CLIP by Learning to Rank Local Patches

TL;DR

This paper introduces a method to efficiently prune patch tokens in CLIP's ViT backbone using a learned ranking system, reducing computation with minimal accuracy loss across multiple datasets.

Contribution

It proposes a novel token pruning framework with a lightweight predictor and learnable tokens, significantly improving CLIP's efficiency while maintaining performance.

Findings

Reduced 40% of patch tokens with only 0.3% accuracy loss

Developed a greedy search for optimal token ranking

Achieved systematic token pruning across seven datasets

Abstract

Contrastive image-text pre-trained models such as CLIP have shown remarkable adaptability to downstream tasks. However, they face challenges due to the high computational requirements of the Vision Transformer (ViT) backbone. Current strategies to boost ViT efficiency focus on pruning patch tokens but fall short in addressing the multimodal nature of CLIP and identifying the optimal subset of tokens for maximum performance. To address this, we propose greedy search methods to establish a "Golden Ranking" and introduce a lightweight predictor specifically trained to approximate this Ranking. To compensate for any performance degradation resulting from token pruning, we incorporate learnable visual tokens that aid in restoring and potentially enhancing the model's performance. Our work presents a comprehensive and systematic investigation of pruning tokens within the ViT backbone of CLIP models. Through our framework, we successfully reduced 40% of patch tokens in CLIP's ViT while only suffering a minimal average accuracy loss of 0.3 across seven datasets. Our study lays the groundwork for building more computationally efficient multimodal models without sacrificing their performance, addressing a key challenge in the application of advanced vision-language models.