CLIP the Divergence: Language-guided Unsupervised Domain Adaptation

TL;DR

This paper introduces CLIP-Div, a novel language-guided approach for unsupervised domain adaptation that leverages CLIP to measure domain divergence and calibrate pseudo labels, significantly improving performance over existing CNN-based methods.

Contribution

The work proposes two new language-guided divergence measurement losses and a pseudo-labeling strategy using CLIP, advancing unsupervised domain adaptation techniques.

Findings

Outperforms state-of-the-art CNN-based UDA methods by large margins.

Achieves +10.3% on Office-Home, +1.5% on Office-31, +0.2% on VisDA-2017, +24.3% on DomainNet.

Demonstrates effective use of CLIP for domain divergence measurement and pseudo-label calibration.

Abstract

Unsupervised domain adaption (UDA) has emerged as a popular solution to tackle the divergence between the labeled source and unlabeled target domains. Recently, some research efforts have been made to leverage large vision-language models, such as CLIP, and then fine-tune or learn prompts from them for addressing the challenging UDA task. In this work, we shift the gear to a new direction by directly leveraging CLIP to measure the domain divergence and propose a novel language-guided approach for UDA, dubbed as CLIP-Div. Our key idea is to harness CLIP to 1) measure the domain divergence via the acquired domain-agnostic distribution and 2) calibrate the target pseudo labels with language guidance, to effectively reduce the domain gap and improve the UDA model's generalization capability. Specifically, our major technical contribution lies in the proposed two novel language-guided domain divergence measurement losses: absolute divergence and relative divergence. These loss terms furnish precise guidelines for aligning the distributions of the source and target domains with the domain-agnostic distribution derived from CLIP. Additionally, we propose a language-guided pseudo-labeling strategy for calibrating the target pseudo labels. Buttressed by it, we show that a further implementation for self-training can enhance the UDA model's generalization capability on the target domain. CLIP-Div surpasses state-of-the-art CNN-based methods by a substantial margin, achieving a performance boost of +10.3% on Office-Home, +1.5% on Office-31, +0.2% on VisDA-2017, and +24.3% on DomainNet, respectively.