Sparse-Tuning: Adapting Vision Transformers with Efficient Fine-tuning and Inference

TL;DR

Sparse-Tuning introduces a novel framework combining token sparsification and dense adapters to efficiently fine-tune Vision Transformers, significantly reducing computational costs while maintaining high performance across image and video tasks.

Contribution

It proposes a new method that enhances PEFT by improving inference efficiency through token sparsification and information compensation with dense adapters.

Findings

Reduces GFLOPs to 66% of original ViT-B

Achieves state-of-the-art performance on multiple datasets

Maintains performance with significantly less computation

Abstract

Parameter-efficient fine-tuning (PEFT) has emerged as a popular solution for adapting pre-trained Vision Transformer (ViT) models to downstream applications by updating only a small subset of parameters. While current PEFT methods have achieved fine-tuning efficiency, they overlook the efficiency of computation and GPU memory during inference, falling short of practical requirements. To address this limitation, we propose Sparse-Tuning, an efficient and effective framework that leverages popular token sparsification (TS) techniques to reduce information redundancy in images and videos, thereby significantly improving computational and memory efficiency. However, TS often compromises performance due to inevitable information loss. To address this limitation, we further introduce Dense Adapters (DA) to compensate for the information losses incurred by token sparsification. DA integrates comprehensive token information from shallow layers into the retained tokens of deeper layers, ensuring minimal performance degradation. Through the integration of TS techniques and DA, Sparse-Tuning achieves a significant reduction in computation and memory overhead while maintaining performance. Empirical results on VTAB-1K, three image datasets, and two video datasets show that Sparse-Tuning reduces GFLOPs to 66\% of the original ViT-B while achieving state-of-the-art performance compared to full fine-tuning and other PEFT baselines.