Open-Set Domain Adaptation with Visual-Language Foundation Models

TL;DR

This paper leverages visual-language foundation models like CLIP for open-set domain adaptation, introducing an entropy optimization strategy that enhances zero-shot prediction and achieves state-of-the-art results on benchmarks.

Contribution

It proposes a novel method to adapt CLIP for open-set domain adaptation, improving performance without extensive fine-tuning.

Findings

CLIP-based zero-shot prediction performs well in ODA tasks

Entropy optimization enhances ODA model accuracy

Achieves state-of-the-art results on multiple benchmarks

Abstract

Unsupervised domain adaptation (UDA) has proven to be very effective in transferring knowledge obtained from a source domain with labeled data to a target domain with unlabeled data. Owing to the lack of labeled data in the target domain and the possible presence of unknown classes, open-set domain adaptation (ODA) has emerged as a potential solution to identify these classes during the training phase. Although existing ODA approaches aim to solve the distribution shifts between the source and target domains, most methods fine-tuned ImageNet pre-trained models on the source domain with the adaptation on the target domain. Recent visual-language foundation models (VLFM), such as Contrastive Language-Image Pre-Training (CLIP), are robust to many distribution shifts and, therefore, should substantially improve the performance of ODA. In this work, we explore generic ways to adopt CLIP, a popular VLFM, for ODA. We investigate the performance of zero-shot prediction using CLIP, and then propose an entropy optimization strategy to assist the ODA models with the outputs of CLIP. The proposed approach achieves state-of-the-art results on various benchmarks, demonstrating its effectiveness in addressing the ODA problem.