Source-Free Domain Adaptation with Frozen Multimodal Foundation Model

TL;DR

This paper introduces DIFO, a novel approach that leverages off-the-shelf vision-language models for source-free domain adaptation by customizing and distilling knowledge to improve target model performance.

Contribution

It proposes a new method combining prompt-based customization and knowledge distillation of multimodal models for source-free domain adaptation.

Findings

DIFO outperforms existing state-of-the-art methods.

Regularization terms improve distillation reliability.

Effective adaptation with off-the-shelf multimodal models.

Abstract

Source-Free Domain Adaptation (SFDA) aims to adapt a source model for a target domain, with only access to unlabeled target training data and the source model pre-trained on a supervised source domain. Relying on pseudo labeling and/or auxiliary supervision, conventional methods are inevitably error-prone. To mitigate this limitation, in this work we for the first time explore the potentials of off-the-shelf vision-language (ViL) multimodal models (e.g.,CLIP) with rich whilst heterogeneous knowledge. We find that directly applying the ViL model to the target domain in a zero-shot fashion is unsatisfactory, as it is not specialized for this particular task but largely generic. To make it task specific, we propose a novel Distilling multimodal Foundation model(DIFO)approach. Specifically, DIFO alternates between two steps during adaptation: (i) Customizing the ViL model by maximizing the mutual information with the target model in a prompt learning manner, (ii) Distilling the knowledge of this customized ViL model to the target model. For more fine-grained and reliable distillation, we further introduce two effective regularization terms, namely most-likely category encouragement and predictive consistency. Extensive experiments show that DIFO significantly outperforms the state-of-the-art alternatives. Code is here