Enhancing Domain Adaptation through Prompt Gradient Alignment

TL;DR

This paper introduces a novel gradient alignment approach for unsupervised domain adaptation using prompt learning, improving the learning of discriminative features and outperforming existing methods.

Contribution

It proposes a multiple-objective gradient alignment framework for UDA with prompt learning, including a gradient norm penalty to prevent overfitting, applicable to single and multi-source scenarios.

Findings

Consistently outperforms existing vision-language model adaptation methods

Effective gradient alignment improves domain adaptation performance

Gradient norm penalty enhances model generalization

Abstract

Prior Unsupervised Domain Adaptation (UDA) methods often aim to train a domain-invariant feature extractor, which may hinder the model from learning sufficiently discriminative features. To tackle this, a line of works based on prompt learning leverages the power of large-scale pre-trained vision-language models to learn both domain-invariant and specific features through a set of domain-agnostic and domain-specific learnable prompts. Those studies typically enforce invariant constraints on representation, output, or prompt space to learn such prompts. In contrast, we cast UDA as a multiple-objective optimization problem in which each objective is represented by a domain loss. Under this new framework, we propose to align per-objective gradients to foster consensus between them. Additionally, to prevent potential overfitting when fine-tuning this deep learning architecture, we penalize the norm of these gradients. To achieve these goals, we devise a practical gradient update procedure that can work under both single-source and multi-source UDA. Empirically, our method consistently outperforms other vision-language model adaptation methods. The implementation is available at https://github.com/VietHoang1512/PGA.