Combining inherent knowledge of vision-language models with unsupervised domain adaptation through strong-weak guidance

TL;DR

This paper presents a novel approach combining unsupervised domain adaptation with the inherent zero-shot capabilities of vision-language models, using a strong-weak guidance scheme to improve cross-domain image classification.

Contribution

It introduces a strong-weak guidance learning scheme that leverages zero-shot predictions and knowledge distillation to enhance unsupervised domain adaptation in vision-language models.

Findings

Outperforms state-of-the-art methods on three benchmarks.

Effective integration of zero-shot predictions improves adaptation.

Ablation studies confirm the contribution of each component.

Abstract

Unsupervised domain adaptation (UDA) tries to overcome the tedious work of labeling data by leveraging a labeled source dataset and transferring its knowledge to a similar but different target dataset. Meanwhile, current vision-language models exhibit remarkable zero-shot prediction capabilities. In this work, we combine knowledge gained through UDA with the inherent knowledge of vision-language models. We introduce a strong-weak guidance learning scheme that employs zero-shot predictions to help align the source and target dataset. For the strong guidance, we expand the source dataset with the most confident samples of the target dataset. Additionally, we employ a knowledge distillation loss as weak guidance. The strong guidance uses hard labels but is only applied to the most confident predictions from the target dataset. Conversely, the weak guidance is employed to the whole dataset but uses soft labels. The weak guidance is implemented as a knowledge distillation loss with (shifted) zero-shot predictions. We show that our method complements and benefits from prompt adaptation techniques for vision-language models. We conduct experiments and ablation studies on three benchmarks (OfficeHome, VisDA, and DomainNet), outperforming state-of-the-art methods. Our ablation studies further demonstrate the contributions of different components of our algorithm.