Domain-Agnostic Mutual Prompting for Unsupervised Domain Adaptation

TL;DR

This paper introduces DAMP, a novel method for unsupervised domain adaptation that mutually aligns visual and textual embeddings using domain-agnostic prompts, improving transfer across complex domain shifts.

Contribution

The paper proposes a domain-agnostic mutual prompting framework that leverages large-scale pre-trained vision-language models for better cross-domain knowledge transfer.

Findings

DAMP outperforms state-of-the-art methods on three UDA benchmarks.

Mutual alignment of visual and textual prompts enhances domain-invariant feature learning.

The approach effectively handles complex domain shifts in unsupervised adaptation.

Abstract

Conventional Unsupervised Domain Adaptation (UDA) strives to minimize distribution discrepancy between domains, which neglects to harness rich semantics from data and struggles to handle complex domain shifts. A promising technique is to leverage the knowledge of large-scale pre-trained vision-language models for more guided adaptation. Despite some endeavors, current methods often learn textual prompts to embed domain semantics for source and target domains separately and perform classification within each domain, limiting cross-domain knowledge transfer. Moreover, prompting only the language branch lacks flexibility to adapt both modalities dynamically. To bridge this gap, we propose Domain-Agnostic Mutual Prompting (DAMP) to exploit domain-invariant semantics by mutually aligning visual and textual embeddings. Specifically, the image contextual information is utilized to prompt the language branch in a domain-agnostic and instance-conditioned way. Meanwhile, visual prompts are imposed based on the domain-agnostic textual prompt to elicit domain-invariant visual embeddings. These two branches of prompts are learned mutually with a cross-attention module and regularized with a semantic-consistency loss and an instance-discrimination contrastive loss. Experiments on three UDA benchmarks demonstrate the superiority of DAMP over state-of-the-art approaches.