Few-shot Fine-tuning is All You Need for Source-free Domain Adaptation

TL;DR

This paper argues that few-shot fine-tuning of pretrained models is a practical and reliable alternative to source-free unsupervised domain adaptation, especially in real-world scenarios with data distribution shifts.

Contribution

It demonstrates that few-shot fine-tuning surpasses existing SFUDA methods under realistic conditions and challenges the belief that limited data causes overfitting.

Findings

Few-shot fine-tuning performs comparably to SFUDA in standard settings.

Few-shot fine-tuning outperforms SFUDA in real-world scenarios with distribution shifts.

Carefully fine-tuned models do not suffer from overfitting with limited data.

Abstract

Recently, source-free unsupervised domain adaptation (SFUDA) has emerged as a more practical and feasible approach compared to unsupervised domain adaptation (UDA) which assumes that labeled source data are always accessible. However, significant limitations associated with SFUDA approaches are often overlooked, which limits their practicality in real-world applications. These limitations include a lack of principled ways to determine optimal hyperparameters and performance degradation when the unlabeled target data fail to meet certain requirements such as a closed-set and identical label distribution to the source data. All these limitations stem from the fact that SFUDA entirely relies on unlabeled target data. We empirically demonstrate the limitations of existing SFUDA methods in real-world scenarios including out-of-distribution and label distribution shifts in target data, and verify that none of these methods can be safely applied to real-world settings. Based on our experimental results, we claim that fine-tuning a source pretrained model with a few labeled data (e.g., 1- or 3-shot) is a practical and reliable solution to circumvent the limitations of SFUDA. Contrary to common belief, we find that carefully fine-tuned models do not suffer from overfitting even when trained with only a few labeled data, and also show little change in performance due to sampling bias. Our experimental results on various domain adaptation benchmarks demonstrate that the few-shot fine-tuning approach performs comparatively under the standard SFUDA settings, and outperforms comparison methods under realistic scenarios. Our code is available at https://github.com/daintlab/fewshot-SFDA .