Universal Domain Adaptation from Foundation Models: A Baseline Study

TL;DR

This paper empirically studies the use of foundation models like CLIP for universal domain adaptation, revealing challenges with fine-tuning and proposing a simple, effective CLIP distillation method that improves performance on benchmark tasks.

Contribution

It provides a comprehensive empirical analysis of foundation models in UniDA and introduces a novel CLIP distillation technique with self-calibration for improved out-class detection.

Findings

Fine-tuning from foundation models often underperforms compared to training from scratch.

Existing UniDA methods do not significantly benefit from foundation models when backbone is frozen.

The proposed CLIP distillation method outperforms previous approaches on multiple benchmarks.

Abstract

Foundation models (e.g., CLIP or DINOv2) have shown their impressive learning and transfer capabilities in a wide range of visual tasks, by training on a large corpus of data and adapting to specific downstream tasks. It is, however, interesting that foundation models have not been fully explored for universal domain adaptation (UniDA), which is to learn models using labeled data in a source domain and unlabeled data in a target one, such that the learned models can successfully adapt to the target data. In this paper, we make comprehensive empirical studies of state-of-the-art UniDA methods using foundation models. We first observe that, unlike fine-tuning from ImageNet pre-trained models, as previous methods do, fine-tuning from foundation models yields significantly poorer results, sometimes even worse than training from scratch. While freezing the backbones, we demonstrate that although the foundation models greatly improve the performance of the baseline method that trains the models on the source data alone, existing UniDA methods generally fail to improve over the baseline. This suggests that new research efforts are very necessary for UniDA using foundation models. Based on these findings, we introduce \textit{CLIP distillation}, a parameter-free method specifically designed to distill target knowledge from CLIP models. The core of our \textit{CLIP distillation} lies in a self-calibration technique for automatic temperature scaling, a feature that significantly enhances the baseline's out-class detection capability. Although simple, our method outperforms previous approaches in most benchmark tasks, excelling in evaluation metrics including H-score/H$^3$-score and the newly proposed universal classification rate (UCR) metric. We hope that our investigation and the proposed simple framework can serve as a strong baseline to facilitate future studies in this field.