Partial Fine-Tuning: A Successor to Full Fine-Tuning for Vision Transformers

TL;DR

This paper introduces Partial Fine-Tuning for vision transformers, which improves efficiency and accuracy by selectively tuning parts of the model, guided by a new metric, and enhances model generalization.

Contribution

It proposes a novel partial fine-tuning approach with a layer selection metric, demonstrating its effectiveness across datasets and models, outperforming full fine-tuning in efficiency and accuracy.

Findings

Partial fine-tuning can outperform full fine-tuning in accuracy.

Selective tuning of layers is crucial for optimal performance.

Partial fine-tuning enhances model generalization and efficiency.

Abstract

Fine-tuning pre-trained foundation models has gained significant popularity in various research fields. Existing methods for fine-tuning can be roughly divided into two categories, namely Parameter-Efficient Fine-Tuning and High-Performance Fine-Tuning. The former aims at improving efficiency, while the latter focuses on enhancing performance. Beyond these methods, we demonstrate that Partial Fine-Tuning can be an innovative and promising direction capable of concurrently enhancing both efficiency and accuracy. We first validate eight manually-defined partial fine-tuning strategies across kinds of datasets and vision transformer architectures, and find that some partial fine-tuning strategies (e.g., ffn only or attention only) can achieve better performance with fewer tuned parameters than full fine-tuning, and selecting appropriate layers is critical to partial fine-tuning. Thus, we propose a novel fine-tuned angle metric to guide the selection of appropriate layers for partial fine-tuning, making it flexible to be adapted to various scenarios for more practicable partial fine-tuning. Additionally, we show that partial fine-tuning can serve as a new dimension for Model Soups, improving both the model performance and generalization with fewer tuned parameters. Comprehensive experiments on a wide range of datasets and models validate the great potential of partial fine-tuning.