Domain Adaptation with a Single Vision-Language Embedding

TL;DR

This paper introduces a novel domain adaptation framework using a single vision-language embedding, enabling effective zero-shot and one-shot adaptation without target data, leveraging contrastive pre-training and style augmentation.

Contribution

The work proposes a new method for domain adaptation that relies on a single VL embedding and a style augmentation technique called PIN, eliminating the need for full target data during training.

Findings

Outperforms relevant baselines in semantic segmentation tasks.

Effective in zero-shot and one-shot domain adaptation scenarios.

Utilizes a single VL embedding for style augmentation and domain adaptation.

Abstract

Domain adaptation has been extensively investigated in computer vision but still requires access to target data at the training time, which might be difficult to obtain in some uncommon conditions. In this paper, we present a new framework for domain adaptation relying on a single Vision-Language (VL) latent embedding instead of full target data. First, leveraging a contrastive language-image pre-training model (CLIP), we propose prompt/photo-driven instance normalization (PIN). PIN is a feature augmentation method that mines multiple visual styles using a single target VL latent embedding, by optimizing affine transformations of low-level source features. The VL embedding can come from a language prompt describing the target domain, a partially optimized language prompt, or a single unlabeled target image. Second, we show that these mined styles (i.e., augmentations) can be used for zero-shot (i.e., target-free) and one-shot unsupervised domain adaptation. Experiments on semantic segmentation demonstrate the effectiveness of the proposed method, which outperforms relevant baselines in the zero-shot and one-shot settings.