Prompt Disentanglement via Language Guidance and Representation Alignment for Domain Generalization

TL;DR

This paper introduces a novel framework for domain generalization that leverages language-guided disentanglement of visual prompts and representation alignment, improving model robustness across unseen domains.

Contribution

It proposes a text feature-guided visual prompt tuning framework combined with Worst Explicit Representation Alignment (WERA) to enhance domain-invariant features in visual models.

Findings

Outperforms state-of-the-art DG methods on multiple datasets

Effective disentanglement of text and visual features improves generalization

Incorporating stylized augmentations enhances domain diversity and robustness

Abstract

Domain Generalization (DG) seeks to develop a versatile model capable of performing effectively on unseen target domains. Notably, recent advances in pre-trained Visual Foundation Models (VFMs), such as CLIP, have demonstrated considerable potential in enhancing the generalization capabilities of deep learning models. Despite the increasing attention toward VFM-based domain prompt tuning within DG, the effective design of prompts capable of disentangling invariant features across diverse domains remains a critical challenge. In this paper, we propose addressing this challenge by leveraging the controllable and flexible language prompt of the VFM. Noting that the text modality of VFMs is naturally easier to disentangle, we introduce a novel framework for text feature-guided visual prompt tuning. This framework first automatically disentangles the text prompt using a large language model (LLM) and then learns domain-invariant visual representation guided by the disentangled text feature. However, relying solely on language to guide visual feature disentanglement has limitations, as visual features can sometimes be too complex or nuanced to be fully captured by descriptive text. To address this, we introduce Worst Explicit Representation Alignment (WERA), which extends text-guided visual prompts by incorporating an additional set of abstract prompts. These prompts enhance source domain diversity through stylized image augmentations, while alignment constraints ensure that visual representations remain consistent across both the original and augmented distributions. Experiments conducted on major DG datasets, including PACS, VLCS, OfficeHome, DomainNet, and TerraInc, demonstrate that our proposed method outperforms state-of-the-art DG methods.