CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity Prediction

TL;DR

CP-ViT introduces a progressive, dynamic pruning framework for Vision Transformers that reduces computational cost by selectively removing less informative patches and heads, maintaining high accuracy.

Contribution

The paper proposes a novel cascade pruning method for ViT models that predicts and applies sparsity dynamically, improving efficiency without significant accuracy loss.

Findings

Reduces over 40% FLOPs by pruning 50% patches.

Maintains accuracy loss within 1% after pruning.

Effective across various datasets and pre-trained ViT models.

Abstract

Vision transformer (ViT) has achieved competitive accuracy on a variety of computer vision applications, but its computational cost impedes the deployment on resource-limited mobile devices. We explore the sparsity in ViT and observe that informative patches and heads are sufficient for accurate image recognition. In this paper, we propose a cascade pruning framework named CP-ViT by predicting sparsity in ViT models progressively and dynamically to reduce computational redundancy while minimizing the accuracy loss. Specifically, we define the cumulative score to reserve the informative patches and heads across the ViT model for better accuracy. We also propose the dynamic pruning ratio adjustment technique based on layer-aware attention range. CP-ViT has great general applicability for practical deployment, which can be applied to a wide range of ViT models and can achieve superior accuracy with or without fine-tuning. Extensive experiments on ImageNet, CIFAR-10, and CIFAR-100 with various pre-trained models have demonstrated the effectiveness and efficiency of CP-ViT. By progressively pruning 50\% patches, our CP-ViT method reduces over 40\% FLOPs while maintaining accuracy loss within 1\%.