SAFT: Towards Out-of-Distribution Generalization in Fine-Tuning

TL;DR

SAFT is a simple yet effective fine-tuning method that updates only a small subset of important parameters to improve out-of-distribution generalization in pre-trained vision-language models like CLIP.

Contribution

SAFT introduces a parameter-efficient fine-tuning approach that preserves pre-trained knowledge while enhancing OOD performance, using only 0.1% of parameters.

Findings

SAFT significantly improves CLIP's OOD performance.

SAFT outperforms baseline fine-tuning methods.

SAFT achieves a 5.15% gain on ImageNet OOD benchmarks.

Abstract

Handling distribution shifts from training data, known as out-of-distribution (OOD) generalization, poses a significant challenge in the field of machine learning. While a pre-trained vision-language model like CLIP has demonstrated remarkable zero-shot performance, further adaptation of the model to downstream tasks leads to undesirable degradation for OOD data. In this work, we introduce Sparse Adaptation for Fine-Tuning (SAFT), a method that prevents fine-tuning from forgetting the general knowledge in the pre-trained model. SAFT only updates a small subset of important parameters whose gradient magnitude is large, while keeping the other parameters frozen. SAFT is straightforward to implement and conceptually simple. Extensive experiments show that with only 0.1% of the model parameters, SAFT can significantly improve the performance of CLIP. It consistently outperforms baseline methods across several benchmarks. On the few-shot learning benchmark of ImageNet and its variants, SAFT gives a gain of 5.15% on average over the conventional fine-tuning method in OOD settings.