Source-Free Domain Adaptation Guided by Vision and Vision-Language Pre-Training

TL;DR

This paper introduces a flexible framework for source-free domain adaptation that leverages pre-trained vision and vision-language models, improving pseudolabel quality and adaptation performance across various challenging scenarios.

Contribution

It proposes an integrated approach to incorporate pre-trained networks, including CLIP, into the SFDA process, enhancing representation and adaptation capabilities.

Findings

Improved target domain adaptation performance on multiple benchmarks.

Effective integration of CLIP for zero-shot classification in SFDA.

Robustness in open-set, partial-set, and open-partial scenarios.

Abstract

Source-free domain adaptation (SFDA) aims to adapt a source model trained on a fully-labeled source domain to a related but unlabeled target domain. While the source model is a key avenue for acquiring target pseudolabels, the generated pseudolabels may exhibit source bias. In the conventional SFDA pipeline, a large data (e.g. ImageNet) pre-trained feature extractor is used to initialize the source model at the start of source training, and subsequently discarded. Despite having diverse features important for generalization, the pre-trained feature extractor can overfit to the source data distribution during source training and forget relevant target domain knowledge. Rather than discarding this valuable knowledge, we introduce an integrated framework to incorporate pre-trained networks into the target adaptation process. The proposed framework is flexible and allows us to plug modern pre-trained networks into the adaptation process to leverage their stronger representation learning capabilities. For adaptation, we propose the Co-learn algorithm to improve target pseudolabel quality collaboratively through the source model and a pre-trained feature extractor. Building on the recent success of the vision-language model CLIP in zero-shot image recognition, we present an extension Co-learn++ to further incorporate CLIP's zero-shot classification decisions. We evaluate on 4 benchmark datasets and include more challenging scenarios such as open-set, partial-set and open-partial SFDA. Experimental results demonstrate that our proposed strategy improves adaptation performance and can be successfully integrated with existing SFDA methods. Project code is available at https://github.com/zwenyu/colearn-plus.