CrAFT: Compression-Aware Fine-Tuning for Efficient Visual Task Adaptation

TL;DR

CrAFT is a simple, efficient fine-tuning framework that enhances post-training compression methods like pruning and quantization, enabling effective deployment of large vision models with minimal additional training.

Contribution

It introduces a novel fine-tuning approach with sharpness minimization that improves compression performance without significant training overhead.

Findings

Significantly boosts pruning and quantization effectiveness.

Achieves task adaptation with minimal fine-tuning time.

Applicable to both convolution and attention-based models.

Abstract

Transfer learning has become a popular task adaptation method in the era of foundation models. However, many foundation models require large storage and computing resources, which makes off-the-shelf deployment impractical. Post-training compression techniques such as pruning and quantization can help lower deployment costs. Unfortunately, the resulting performance degradation limits the usability and benefits of such techniques. To close this performance gap, we propose CrAFT, a simple fine-tuning framework that enables effective post-training network compression. In CrAFT, users simply employ the default fine-tuning schedule along with sharpness minimization objective, simultaneously facilitating task adaptation and compression-friendliness. Contrary to the conventional sharpness minimization techniques, which are applied during pretraining, the CrAFT approach adds negligible training overhead as fine-tuning is done in under a couple of minutes or hours with a single GPU. The effectiveness of CrAFT, which is a general-purpose tool that can significantly boost one-shot pruning and post-training quantization, is demonstrated on both convolution-based and attention-based vision foundation models on a variety of target tasks. The code will be made publicly available.