Rethinking the Need for Source Models: Source-Free Domain Adaptation from Scratch Guided by a Vision-Language Model

TL;DR

This paper introduces a new source-free domain adaptation setting that relies solely on a vision-language model and unlabeled target data, eliminating dependence on source models, and proposes a two-stage framework that achieves competitive results.

Contribution

The paper proposes VODA, a stricter source-free domain adaptation setting, and introduces TS-DRD, a two-stage method that effectively guides adaptation without source models.

Findings

TS-DRD achieves competitive performance on multiple benchmarks.

VODA setting demonstrates that source models have limited impact on adaptation.

The method effectively utilizes vision-language guidance for domain adaptation.

Abstract

Source-Free Domain Adaptation (SFDA) adapts source models to target domains without accessing source data, addressing privacy and transmission issues. However, existing methods still initialize from a source pre-trained model and thus are not truly source-free. Recent works have introduced Vision-Language (ViL) models to guide the adaptation process, in these methods, we observe that for the same target domain, different source models yield minimal variation in final results, indicating the source model itself has limited impact. Motivated by this, we propose ViL-Only Domain Adaptation (VODA) , a stricter setting that eliminates all dependencies on source domain, relying solely on a randomly initialized model, a ViL model, and unlabeled target data. We analyze the adaptation dynamics of VODA and introduce Two-Stage Denoised-Region Distillation (TS-DRD) , a two-stage framework that first warms up the model with ViL guidance, then seek a Denoised-Region inherent in both the ViL and adapting model, yielding cleaner supervision for distillation. Experiments on Office-Home, VisDA, and DomainNet-126 show that under VODA, TS-DRD achieves competitive or superior performance to existing SFDA methods that still use source models, demonstrating its effectiveness and the potential of the VODA setting.